search

AAndyProgram / SCrawler

🏳️‍🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.
https://discord.gg/uFNUXvFFmg
GNU General Public License v3.0
1.23k stars 81 forks source link

Failed to crawl all Reddit user files #47

Closed tayser1 closed 2 years ago

tayser1 commented 2 years ago

Doesn't download all reddit user files

Example URL (NSFW): https://old.reddit.com/user/exhale_jay

Version: Latest version 3.0.0.9

AAndyProgram commented 2 years ago

Is any data on the LOG? Which post is not downloaded? URL SCrawler only downloads 1000 posts. This is the Reddit limit, not mine.

tayser1 commented 2 years ago

No LOG data available (No Error is shown) Example post with image that is not downloaded: https://old.reddit.com/r/gamingsetups/comments/ulwt9w/its_still_coming_together_but_peep_the_zowie_pink/

AAndyProgram commented 2 years ago

Where did you find that crazy chick? 🤢

I downloaded the post you pointed to. image

tayser1 commented 2 years ago

Ok, I deleted the user from Scrawler and re-downloaded and no issue.

So now the question, do I have to manually check every user for completion? What happened: I used the usual download option and not all files got downloaded, which was weird. I checked and stumbled upon reddit user posts that got not downloaded. I used the user download option to update user files and crawl for posts. But the missing posts didn't get downloaded. I deleted the user with data from Scrawler and redownloaded, and it was able to download all posts.

Suggestion: maybe add an option to check user for error/completion

AAndyProgram commented 2 years ago

Ok, I deleted the user from Scrawler and re-downloaded and no issue.

Glad everything is working.

This is the first time I see such a problem. It's possible that Reddit hasn't been available for some time while you were downloading. Didn't SCrawler notify you that Reddir was unavailable before the download started?

So now the question, do I have to manually check every user for completion?

I don't think so. I think this is an accidental error.

I used the user download option to update user files and crawl for posts. But the missing posts didn't get downloaded.

SCrawler does not have this option. The only way is to redownload user. To do this, you need to delete ....._Data.xml and ....._Posts.txt, restart SCrawler and download it again.

maybe add an option to check user for error/completion

How do you propose to do this?

AAndyProgram commented 2 years ago

Since I didn't receive your reply, I consider this issue resolved.

If you like SCrawler, please like the program on this site: https://alternativeto.net/software/scrawler/about/.