search

AAndyProgram / SCrawler

🏳️‍🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.
https://discord.gg/uFNUXvFFmg
GNU General Public License v3.0
1.21k stars 81 forks source link

Incompletely found posts from ThisVid users, blank entries on missing posts #147

Closed pingzoo closed 1 year ago

pingzoo commented 1 year ago

https://thisvid.com/members/1169929/

User has: 91 public videos 53 private videos 1 album with 7 images

SCrawler found 48 public videos (page 1 and 2). SCrawler downloaded 45 public videos -> Log sais "There are missing posts". Missing posts form for this user shows: image I can't select a missing post here.

When I start download again SCrawler sais "All data downloaded".

What should I do in such a case?

I deleted the *_Posts.txt and restarted the download. SCrawler found 60 Posts (53 private + 7 images) and started downloading (currently running).

So after the download at least 46 public videos will be missing again.

I'll keep you posted, when the current download job is done.

AAndyProgram commented 1 year ago

Not confirmed. Look, I added this profile, SCrawler found 91 public videos (like on the site). This site is too slow. I've already downloaded 46 videos in 5 hours. I canceled the download because it's too long.

image

image

pingzoo commented 1 year ago

Hmm ok. Don't know why it happens on my end. But in the last days I got more often error messages, on ThisVid, because of overload of the server (Bad Gateway, 502, 503, etc) Maybe this happend, when SCrawler curling/invoking the site?

pingzoo commented 1 year ago

The Download of 53 private and 7 images is done now. When i start the download again, SCrawler instantly responds with notification "All data downloaded"

I deleted Posts.txt again and startet the download again. SCrawler now found the missing 46 public videos and startet downloading. (job currently running) ;D

pingzoo commented 1 year ago

Ok. Thank you again for checking this on your end. So I guess, it's because of thisvids server load.

Is there any other way than deleting the Posts.txt and starting the download again, or is that ok in this case?

AAndyProgram commented 1 year ago

But in the last days I got more often error messages, on ThisVid, because of overload of the server (Bad Gateway, 502, 503, etc)

Indeed. This is not my main site, but even I have seen these messages.

Maybe this happend, when SCrawler curling/invoking the site?

If the log is empty, then SCrawler didn't face these errors. To get the video download URL, SCrawler uses yt-dlp. Don't forget the cookies!

The Download of 53 private and 7 images is done now. When i start the download again, SCrawler instantly responds with notification "All data downloaded"

I deleted Posts.txt again and startet the download again. SCrawler now found the missing 46 public videos and startet downloading. (job currently running) ;D

Different numbers mean that something is wrong on your end. Try updating your cookies, then add this profile and download it separately. If you're using automation, pause it. Don't start downloading any other profiles until this one has finished downloading.

Ok. Thank you again for checking this on your end.

You're welcome

So I guess, it's because of thisvids server load.

Perhaps

Is there any other way than deleting the Posts.txt and starting the download again, or is that ok in this case?

No, there are no other ways. It's okay.