search

cuemacro / findatapy

Python library to download market data via Bloomberg, Eikon, Quandl, Yahoo etc.
Apache License 2.0
1.7k stars 206 forks source link

limit the threads to the number of CPUs #3

Open spyamine opened 7 years ago

spyamine commented 7 years ago

I have only 4 CPU and when I use 8 for bloomberg it just stopped working!! this limitation fixed the problem for me. We can solve the problem more elegantly by putting the limit in the parameters file.

saeedamen commented 7 years ago

Ah, I see. I probably want to leave some flexibility though around this parameter, so people can adjust to what's most appropriate for their machine. The main delay from downloading is the IO blocking, so theoretically you can set a higher number of threads than CPUs (strictly speaking using the thread model in Python is not truly multithreading because still has GIL).

spyamine commented 7 years ago

ok thanks for the info

2016-11-28 15:06 GMT+00:00 Saeed Amen notifications@github.com:

Ah, I see. I probably want to leave some flexibility though around this parameter, so people can adjust to what's most appropriate for their machine. The main delay from downloading is the IO blocking, so theoretically you can set a higher number of threads than CPUs (strictly speaking using the thread model in Python is not truly multithreading because still has GIL).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cuemacro/findatapy/pull/3#issuecomment-263294334, or mute the thread https://github.com/notifications/unsubscribe-auth/AMd0fZ2oZ4_grtUlaTQqxJaKBKFAlCrxks5rCu3ogaJpZM4K93DX .