search

AAndyProgram / SCrawler

🏳️‍🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.
https://discord.gg/uFNUXvFFmg
GNU General Public License v3.0
1.35k stars 90 forks source link

only downloading 800 posts on twitter #155

Closed Mas99 closed 1 year ago

Mas99 commented 1 year ago

Hi,

Not sure if it is my setting but SC is only downloading the 1st 800 posts on twitter ? How do I change it ?

AAndyProgram commented 1 year ago

There is nothing to change. Twitter has always been limited to parsing only the top 1000 posts. Currently twitter has gone completely crazy. So the fact that you downloaded 800 posts is already good.

sriwijaya572 commented 4 months ago

Hi @AAndyProgram is there any way to continue download past 1k media? like there will be countdown every 500 download, depreciated extension "Twitter media downloader" use this method to bypass it that time. but eventually twitter api become paid and the extension got depreciated by them

AAndyProgram commented 4 months ago

@sriwijaya572 no SCrawler uses gallery-dl to download twitter. You should ask there.

Join our Discord server: https://discord.gg/uFNUXvFFmg

sriwijaya572 commented 4 months ago

okay thanks

edit: they already add sleep feature, but i cant find this setting in Scrawler https://github.com/mikf/gallery-dl/issues/5860

krystian2160 commented 1 month ago

As sriwajaya572 said, https://github.com/mikf/gallery-dl/issues/5860 + they added countdown into gallery-dl. Jdownloader downloads whole twitter profile with the right timeout, eg 20.000 ms for every request. Can SCrawler download properly from twitter too? ❤️

AAndyProgram commented 1 month ago

@krystian2160, @sriwijaya572, guys, did you read this issue carefully? As @mikf said:

--sleep and --sleep-request alone are not enough to keep Twitter from suspending your account it seems, but I also can't tell you what is.

krystian2160 commented 1 month ago

So you mean it doesnt work cuz you did not figure out how to do it because sleep is not good enough alone so it works?

AAndyProgram commented 1 month ago

So you mean it doesnt work cuz you did not figure out how to do it because sleep is not good enough alone so it works?

I didn't add it because it might lead to account ban

krystian2160 commented 1 month ago

Might But with good sleep time... JDownloader managaes somehow full crawl with no ban. But I had to do in settings 20.000 ms for every request, so there is no short-time ban (throttle or something).

AAndyProgram commented 1 month ago

I'm willing to consider adding this feature if someone provides me with a Twitter account to tests. Sorry, but I won't debug features that could get my real account banned. 😉