search

4pr0n / ripme

Downloads albums in bulk
MIT License
917 stars 203 forks source link

[Deviantart] It´s possible to speed up the process of download? #584

Closed joselpg closed 7 years ago

joselpg commented 7 years ago

I think the process is too low when we have a lot of images. In my case it starts downloading 24 per page, and delay 3 minutes for start with the next page. So just imagine download a big collection

Hrxn commented 7 years ago

Download speed is always limited by the server, so I don't think there is much that can be done here..

rautamiekka commented 7 years ago

@Hrxn is right, the speed you get with DA is very variable. You can try 2 threads (I wouldn't try more than that, but that's me) since it's a little faster even if DA's end is busy; I've done many folders which have had hundreds of items each, so I know from experience.

If you drop the timeout to 12000 milliseconds, meaning 12 seconds, you might get some boost, at least I seem to have, versus 60000 (60s) or even bigger, where it took even 15 minutes between pages. Seemingly that timeout is how long RipMe waits for replies and re-tries when nothing happens, which'd explain why it takes so long.

The bottom line: don't try too many threads and too low timeouts cuz although DA still hasn't IP-banned me for ripping tons of stuff, they could get wise anytime and suddenly put a range of rippers behind a firewall, which could end uglily. 2 threads and 12s sound like the reasonable max to me to keep using.

joselpg commented 7 years ago

@Hrxn You´re right. Sound logical for me. @rautamiekka Yes. Is configured by default with: 5 Maximum threads and 60000 milliseconds of timeout. I didn't use your recommended configuration. But I tried with a similar, and there isn´t a significant diference for speed up the velocity. I´m donwloading with 24 threads per page, I think is the max. capacity for DA use. Because only 24 images are hosted per page. And about timeout. I´m not sure, but I used 10000 ms. In short, There´s not much you can do about the spped up. I have 20Mbps, and according to my calculations. It will take me for over 30 hours to download 17000 images of my collection. Although with a speed significantly much slower it might take the same.

rautamiekka commented 7 years ago

^ 24 threads is just asking for trouble, from both DA and your connection, and if you've 20Mbps down you're not giving it much per thread.

metaprime commented 7 years ago

Looks like this issue is addressed as "by design". Closing as client-side / by-design / wontfix.