search

NiiMiyo / EDOPro-Hd-Downloader

Downloads HD cards arts images for ProjectIgnis: EDOPro
MIT License
41 stars 4 forks source link

Download multiple images at once #6

Closed Teddy-Kun closed 1 year ago

Teddy-Kun commented 1 year ago

I thought that the download progress could be sped up by doing what the title says. I have had good results so far, but sadly due to how Threading works we get a lot of overhead while checking what is already downloaded. I will move that into a new method soon. Wanted to create this pull request beforehand to see what you think of it.

Teddy-Kun commented 1 year ago

Was easier than expexted lol. Stuff to consider would be putting the thread stuff in loops and letting the user control it with a variable, but I think most wouldn't even understand what the benefit is.

Also I should probably explain why I only use 3 simultaneous downloads: I don't want to DDOS ygoprodeck lol. If you don't think thats a concern let me know

Teddy-Kun commented 1 year ago

Noticed an issue, it doesn't save the progress anymore, will probably fix it tomorrow

NiiMiyo commented 1 year ago

Hi! Thanks for the time to make this edit. Unfortunately, I won't merge your Pull Request.

I have already considered the idea of using threads or async functions to download multiple cards simultaneously but YGOPPRODeck's API Guide states "If we find you are pulling a very high volume of images per second then your IP will be blacklisted and blocked.".

Most people will download all cards (over 12k requests) and since they doesn't specify what is considered a "very high volume of images" and I don't want to ban anyone from the website or the API I used a very low frequency to download cards (about 2 per second).

But you can still fork the repository and keep working on your own version without problems! Just remember to change UserAgent and put your name on the LICENSE

Now, if you are willing to go back and change your code, maybe a different approach would be better than using 3 threads on 3 lists. Some download handler class that uses async calls, a queue and some Observer-like programming pattern would probably be more maintainable and readable.