search

jklmli / manga_downloader

Cross-platform, multi-site, multi-threaded manga downloader with over 5000 distinct mangas. Includes support for automated downloading via external .xml file and conversion for viewing on the Kindle.
MIT License
270 stars 53 forks source link

Use tempfile to allow for multiple instances of script to be run #12

Closed jklmli closed 13 years ago

jklmli commented 13 years ago

The script is currently unable to download more than one manga at a time. The obvious workaround (for now) is to run multiple instances of the program.

Currently the temporary download directory is mangadl_tmp. This causes issues when calls to cleanTmp and compress are made. We could easily use tempfile.mkdtemp, this would allow for different temporary directories, which should fix these inconsistencies.

spikepmw commented 13 years ago

You can download multiple manga at a time using the XML File - Apparently I am pushing the xml feature =). However this method downloads it sequentially, not at the same time.

If the temporary directories is fixed, the script could be written to download the different manga in parallel.

spikepmw commented 13 years ago

Checked into the dev branch (0.7.9)

spikepmw commented 13 years ago

Added Parallel processing for XML File; however, I only got minor improvement. It appears that urllib serializes request. The only speed-up I seemed to get is when the script was converting some images and downloading others.

jklmli commented 13 years ago

Could you please clarify what you mean by 'urrlib serializing requests' and how this relates to speedups?

With regards to the speed, I guess it's not really that important. I just thought the functionality of being able to download two mangas at the same time would be useful.

Maybe we should add an option --sequential or --parallel?

spikepmw commented 13 years ago

I going to try to explain what I mean with a fictitious example.

Lets make the following assumptions (All number are made up to express my point):

  1. Server takes 1 second to respond with an image.
  2. The image takes a fraction of the bandwidth to download

Therefore sequentially to download 2 chapters of different manga with 20 images each would take 40 seconds. (20 * 1 second * 2)

If I could download the chapters at the same time I should see the following speedup

Time=0: Request First Page of 1st Chapter Time=0.1 Request First Page of 2nd Chapter Time=1 Receive First Page of 1st Chapter Time=1.1 Receive Second Page of 2nd Chapter ect...

Therefore the theoretically download time should be around 20.1 seconds.

However on the Windows machine it APPEARED (not measurement done) that there was little speed-up. It APPEARED that running 2 instance of the program that downloaded one chapter was faster than running one that downloaded the 2.

On this appearance I theorized maybe something in the urllib module was serializing request.

However, (On a OSX Machine) since I wrote that I used the time functionality to measure the time and it was slightly faster to run 1 instance that downloads the 2 different mangas.

jklmli commented 13 years ago

Ah, ok. I always just assumed that servers handled requests sequentially.