Closed tayser1 closed 2 years ago
Is any data on the LOG? Which post is not downloaded? URL SCrawler only downloads 1000 posts. This is the Reddit limit, not mine.
No LOG data available (No Error is shown) Example post with image that is not downloaded: https://old.reddit.com/r/gamingsetups/comments/ulwt9w/its_still_coming_together_but_peep_the_zowie_pink/
Where did you find that crazy chick? 🤢
I downloaded the post you pointed to.
Ok, I deleted the user from Scrawler and re-downloaded and no issue.
So now the question, do I have to manually check every user for completion? What happened: I used the usual download option and not all files got downloaded, which was weird. I checked and stumbled upon reddit user posts that got not downloaded. I used the user download option to update user files and crawl for posts. But the missing posts didn't get downloaded. I deleted the user with data from Scrawler and redownloaded, and it was able to download all posts.
Suggestion: maybe add an option to check user for error/completion
Ok, I deleted the user from Scrawler and re-downloaded and no issue.
Glad everything is working.
This is the first time I see such a problem. It's possible that Reddit hasn't been available for some time while you were downloading. Didn't SCrawler notify you that Reddir was unavailable before the download started?
So now the question, do I have to manually check every user for completion?
I don't think so. I think this is an accidental error.
I used the user download option to update user files and crawl for posts. But the missing posts didn't get downloaded.
SCrawler does not have this option. The only way is to redownload user. To do this, you need to delete ....._Data.xml
and ....._Posts.txt
, restart SCrawler and download it again.
maybe add an option to check user for error/completion
How do you propose to do this?
Since I didn't receive your reply, I consider this issue resolved.
If you like SCrawler, please like the program on this site: https://alternativeto.net/software/scrawler/about/.
Doesn't download all reddit user files
Example URL (NSFW): https://old.reddit.com/user/exhale_jay
Version: Latest version 3.0.0.9