AAndyProgram
commented
1 year ago About the last download date. Confirmed. There is only a display bug. It doesn't affect the download in any way. The internal values still store the date, but there was one parameter missing when the new plan was created, that prevented the date from being saved. However, after restarting SCrawler everything works fine. This bug will be fixed in the next version.
About the progress hang. Actually, there is a bug in the latest version. It has to do with the visualization of the user queue. Because of this bug, after downloading the last user, the function did not complete because the download event was stuck in the user queue form. This is the first time I see more than one user hanging. However, you just need to click the Stop button to complete the task. I can't confirm the missing data, probably because I've already fixed the bug. I downloaded r/cats at 1 minute intervals, and each time several images were downloaded.
Join our Discord server: https://discord.gg/uFNUXvFFmg If you like SCrawler, please like the program on this site: https://alternativeto.net/software/scrawler/about/.
KingMissile
Describe the bug I have an automation rule to scrape 18 Reddit profiles every 10 minutes. It appears to run fine the first time when setting a schedule but then unexplainably stops.
To Reproduce Set a rule as shown in Automation:
Click Ok and Start. Scheduler shows the following:
Main SCrawler window shows this in the status bar (no new files were downloaded in this example case):
Scheduler changes to the following:
Then I proceed to leave it running in the background for several hours. In that time the scheduler time/date never updates, the SCrawler main window perpetually says [Jobs 18] All data downloaded and never changes, even when new images are posted to profiles (confirmed myself, no hash de-duplication happening). Clicking Start or Start (force) in the scheduler window doesn't appear to do anything, not even change the next run time. Nothing is accidentally paused. The Stop button in the main window is illuminated while the schedule is "running", but clicking that just seems to reset the next run time to the intended timeframe. When/if it it does run again, it's back to square one.
Restarting the program completely and manually doing the Download all step grabs any/all new images posted in that time as intended (confirmed in the Info (download summary) window). There are no errors in the log to post unfortunately.
Expected behavior If not clear from the above text, I would expect that the program would automatically check each profile every 10 minutes and fetch new data without any intervention on my behalf.
Screenshots As above.
Release information:
SCrawler v2023.8.6.0 (x64) OS: Microsoft Windows 10 Enterprise (x64) ffmpeg version: ffmpeg version 5.1.2-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers yt-dlp version: 2023.07.06 gallery-dl version: 1.25.8 cURL version: curl 7.86.0 (x86_64-w64-mingw32) libcurl/7.86.0 OpenSSL/3.0.7 (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 WinIDN libssh2/1.10.0 nghttp2/1.50.0 ngtcp2/0.10.0 nghttp3/0.7.1 libgsasl/2.2.0
Additional context If this is a user problem (likely) any guidance would be appreciated. I appreciate your time.