ValueRaider / yfinance-cache

Caching wrapper for yfinance module. Intelligent caching, not dumb caching of web requests.
MIT License
22 stars 9 forks source link

AssertionError: daemonic processes are not allowed to have children #43

Closed ana-jiangR closed 1 week ago

ana-jiangR commented 8 months ago

bulk download failure occurred when setting "threads=True" default

see my code below: def test_yfc_download(): tickers = ["INTC", "MSFT"]

  df = yfc.download(tickers, 
              interval='1h', 
              period=TIME_PERIOD_60M, 
              max_age='4h', 
              group_by='ticker', 
              # threads=False
              )

  print(df['MSFT'][['High', 'Low', 'Close', 'Volume']])

  return

if __name__ == '__main__':

  import yfinance_cache as yfc

  TIME_PERIOD_60M = '250d'
  test_yfc_download()`

The failure log is as below:

Process SpawnPoolWorker-3: Traceback (most recent call last): File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 315, in _bootstrap self.run() Traceback (most recent call last): File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 108, in run self._target(*self._args, *self._kwargs) File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 315, in _bootstrap self.run() File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 108, in run self._target(self._args, **self._kwargs) File "C:\miniconda3\envs\yfinance\lib\multiprocessing\pool.py", line 114, in worker task = get() File "C:\miniconda3\envs\yfinance\lib\multiprocessing\pool.py", line 114, in worker task = get() File "C:\miniconda3\envs\yfinance\lib\multiprocessing\queues.py", line 367, in get return _ForkingPickler.loads(res) File "C:\miniconda3\envs\yfinance\lib\multiprocessing\queues.py", line 367, in get return _ForkingPickler.loads(res) File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache__init__.py", line 4, in from .yfc_ticker import Ticker, verify_cached_tickers_prices File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache__init__.py", line 4, in from .yfc_ticker import Ticker, verify_cached_tickers_prices File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_ticker.py", line 7, in from . import yfc_time as yfct File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_ticker.py", line 7, in from . import yfc_time as yfct File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_time.py", line 22, in manager = Manager() File "C:\miniconda3\envs\yfinance\lib\multiprocessing\context.py", line 57, in Manager m.start() File "C:\miniconda3\envs\yfinance\lib\multiprocessing\managers.py", line 554, in start self._process.start() File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 118, in start assert not _current_process._config.get('daemon'), \ File "C:\miniconda3\envs\yfinance\lib\site-packages\yfinance_cache\yfc_time.py", line 22, in manager = Manager() File "C:\miniconda3\envs\yfinance\lib\multiprocessing\context.py", line 57, in Manager m.start() AssertionError: daemonic processes are not allowed to have children File "C:\miniconda3\envs\yfinance\lib\multiprocessing\managers.py", line 554, in start self._process.start() File "C:\miniconda3\envs\yfinance\lib\multiprocessing\process.py", line 118, in start assert not _current_process._config.get('daemon'), \ AssertionError: daemonic processes are not allowed to have children

I'm using Window 11 OS.

my Python version: Python 3.9.18 (main, Sep 11 2023, 14:09:26) [MSC v.1916 64 bit (AMD64)] on win32

see dependences' version: yfance_cache (0.4.7) yfance (0.2.31) pandas (2.1.1) exchange-calendars (4.5) scipy (1.10.1) click (8.1.7) numpy (1.26.0) pyluach (2.2.0) python-dateutil (2.8.2) toolz (0.12.0) tzdata (2023.3) korean-lunar-calendar (0.3.1) pytz (2023.3.post1) requests (2.31.0) multitaskg(0.0.11) lxml (4.9.3) appdirs (1.4.4) frozendict(2.3.8) peewee (3.16.3) beautifulsoup4 (4.12.2) html5lib (1.1) colorama (0.4.4) soupsieve>1.2 (2.5) six>=1.9 (1.16.0) webencodgs (0.5.1) charset-normalizer (3.3.0) idna(3.4) urllib3 (1.26.17) certifi (2023.7.22)

ValueRaider commented 8 months ago

I never had the chance to test on Windows. Turns out multiprocessing behaves differently. I think I found fix but will need you to test.

ana-jiangR commented 8 months ago

Great, thank you. let me know how to test that when your patch is ready.

ana-jiangR commented 8 months ago

Moving discussion to pull request.

ValueRaider commented 8 months ago

Moving discussion to pull request.