search

diogogmt / humblebundle-downloader

Download book from Humble Bundles
MIT License
31 stars 3 forks source link

Download of large files blows memory usage and new download type issues #5

Open affinityv opened 5 years ago

affinityv commented 5 years ago

I've done work to fix this, will do pull request when I have the time. Essentially the changes I made this time allow for download directly to file system files, rather than a string in memory. The sha1 and md5 checks are also done against the files only.

There were some issues with a new bundle [1] , very large video zip files for starters and "supplement" files -- both of which are just zip files, so a bit of fudging for the file naming.

Attached is a quick "diff -u" patch file .... FYI (before the pull request), feel free to incorporate early if you don't want to wait for the pull request. I may make some other minor adjustments as well, but I think it is okay as it is now.

affinityv.patch.txt

[1] https://www.humblebundle.com/books/python-packt-2019-books

tlc commented 4 years ago

Thanks for the downloader, especially the file renaming! This will improve my workflow a lot.

But a large comics download is crawling and using all of my 12G memory. The files so far range from 79M to 795M. I assume the program is trying to download all at once?

If the affinityv pull isn't going to be taken, perhaps a way to limit concurrent downloads?

BTW, at least once I got a comic from HB > 2G.

affinityv commented 4 years ago

I am happy for my code to be fully implemented in to the original project, but I'm not personally able, at this time, to do what needs to be done to get it sorted. People are welcome to diff my code changes and implement it back to the project properly.