search

MalloyDelacroix / DownloaderForReddit

The Downloader for Reddit is a GUI application with some advanced features to extract and download submitted content from reddit.
GNU General Public License v3.0
511 stars 48 forks source link

Decoupled thread count from number of cores #90

Closed zacker150 closed 4 years ago

zacker150 commented 4 years ago

On Windows, QtCore.QThread.idealThreadCount() simply returns the number of logical cores on your computer. However, downloading things is an IO-bound, not CPU bound task. In particular, it is limited by the TCP bandwidth-delay-product, which is why download accelerators demonstrate a speedup with multi-threaded downloading.

In my experiments, I found that increasing the number of threads from 8 to 64 beyond the number of CPU cores increased my mid-extraction bandwidth utilization from 60 Mbps to 99.5 Mbps on a 100 Mbps Ethernet connection when downloading from Eurome.

MalloyDelacroix commented 4 years ago

Very nice. This is a great improvement.