search

earthobservations / wetterdienst

Open weather data for humans.
https://wetterdienst.readthedocs.io/
MIT License
358 stars 55 forks source link

multiprocessing error / segmentation fault #488

Closed AlexDo1 closed 2 years ago

AlexDo1 commented 3 years ago

Describe the bug Hi, I would like to use the package wetterdienst to load data from the DWD server into my Jupyter notebook. For the first few days, downloading the data worked fine (with UnplickingErrors for some parameter-resolution constellations). Since a few days my python kernel crashes every time I try to extract the values from a DwdObservationRequest.

If just running the example code from your documentation in an ipython console, the kernel crashes with a segmentation fault / multprocessing error when I extract the values with df = request.values.all().df.dropna().

The strange thing is that the package worked for the first few days until the error suddenly appeared.

To Reproduce This is the example code from the documentation executed in an ipython console:

❯ ipython
Python 3.9.6 (default, Jul 30 2021, 09:31:09)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from wetterdienst.provider.dwd.observation import DwdObservationRequest,
   ...:  DwdObservationDataset, DwdObservationPeriod, DwdObservationResolution

In [2]: request = DwdObservationRequest(
   ...:         parameter=[DwdObservationDataset.CLIMATE_SUMMARY],
   ...:         resolution=DwdObservationResolution.DAILY,
   ...:         start_date="1990-01-01",
   ...:         end_date="2020-01-01",
   ...:         tidy=True,
   ...:         humanize=True,
   ...:     ).filter_by_name('Rheinstetten')

In [3]: df = request.values.all().df.dropna()
[1]    18409 segmentation fault  ipython
/Users/alexd/opt/anaconda3/envs/mc_develop/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Expected behavior The data for the station 'Rheinstetten' and parameter and resolution specified in DwdObservationRequest should be downloaded and saved to df.

Desktop:

Additional context Do you already now about this bug? A possible solution to this problem could be to use the package single threaded, if that is possible.

gutzbenj commented 3 years ago

Dear @AlexDo1 ,

so far I have only seen a segmentation error when installing scipy under mac m1.

Edit:

I have set up a patch branch with removed threading so that you can install wetterdienst from this branch and run the script another time to test if that works ok?

Install via

pip install git+https://github.com/earthobservations/wetterdienst.git@patch-threading
AlexDo1 commented 3 years ago

Thank you @gutzbenj ,

I just re-ran the code in the To Reproduce section above. I have installed wetterdienst via pip install git+https://github.com/earthobservations/wetterdienst.git@patch-threading

Unfortunately the error remains the same:

In [5]: df = request.values.all().df.dropna()
[1]    22368 segmentation fault  ipython
/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
gutzbenj commented 3 years ago

Dear @AlexDo1,

I still got scipy 1.6.1 running so maybe thats the difference.

The libraries I've installed are

Package Version

aenum 3.1.0 alabaster 0.7.12 anyio 3.3.0 appdirs 1.4.4 appnope 0.1.2 argon2-cffi 20.1.0 attrs 20.3.0 Babel 2.9.1 backcall 0.2.0 bandit 1.7.0 beautifulsoup4 4.9.3 beniget 0.3.0 bitstring 3.1.9 black 20.8b1 bleach 4.0.0 Brotli 1.0.9 cachetools 4.2.2 certifi 2021.5.30 cffi 1.14.6 cfgrib 0.9.9.0 cftime 1.5.0 charset-normalizer 2.0.4 click 7.1.2 click-params 0.1.2 cloudpickle 1.6.0 cloup 0.8.2 colorama 0.4.4 coverage 5.5 crate 0.25.0 css-html-js-minify 2.5.5 cycler 0.10.0 Cython 0.29.23 dash 1.21.0 dash-bootstrap-components 0.12.2 dash-core-components 1.17.1 dash-html-components 1.1.4 dash-table 4.12.0 dask 2021.7.1 dateparser 1.0.0 debugpy 1.4.1 decorator 5.0.9 defusedxml 0.7.1 deprecation 2.1.0 dictdiffer 0.9.0 docutils 0.16 dogpile.cache 1.1.3 duckdb 0.2.7 entrypoints 0.3 et-xmlfile 1.1.0 fastapi 0.61.2 findlibs 0.0.2 flake8 3.9.2 flake8-bandit 2.1.2 flake8-black 0.2.3 flake8-bugbear 20.11.1 flake8-isort 4.0.0 flake8-polyfill 1.0.2 flakehell 0.7.1 Flask 2.0.1 Flask-Compress 1.10.1 freezegun 1.1.0 fsspec 2021.7.0 future 0.18.2 gast 0.4.0 GDAL 3.3.1 geojson 2.5.0 gitdb 4.0.7 GitPython 3.1.20 h11 0.12.0 h5netcdf 0.11.0 h5py 3.3.0 idna 3.2 imagesize 1.2.0 importlib-resources 5.2.2 influxdb 5.3.1 influxdb-client 1.19.0 iniconfig 1.1.1 ipykernel 6.0.3 ipython 7.25.0 ipython-genutils 0.2.0 ipywidgets 7.6.3 isort 5.9.3 itsdangerous 2.0.1 jedi 0.18.0 Jinja2 3.0.1 jsonschema 3.2.0 jupyter 1.0.0 jupyter-client 6.2.0 jupyter-console 6.4.0 jupyter-core 4.7.1 jupyter-server 1.10.2 jupyter-server-mathjax 0.2.3 jupyterlab-widgets 1.0.0 kiwisolver 1.3.1 livereload 2.6.3 locket 0.2.1 lxml 4.6.3 MarkupSafe 2.0.1 matplotlib 3.4.2 matplotlib-inline 0.1.2 mccabe 0.6.1 measurement 3.2.0 mistune 0.8.4 mock 4.0.3 mpmath 1.2.1 msgpack 1.0.2 mypy-extensions 0.4.3 mysqlclient 2.0.3 nbconvert 5.6.1 nbdime 3.1.0 nbformat 5.1.3 nest-asyncio 1.5.1 netCDF4 1.5.7 networkx 2.5.1 notebook 6.4.2 numpy 1.21.1 openpyxl 3.0.7 packaging 21.0 pandas 1.3.1 pandocfilters 1.4.3 parso 0.8.2 partd 1.2.0 pastel 0.2.1 pathspec 0.9.0 pbr 5.6.0 pdbufr 0.9.0 percy 2.0.2 pexpect 4.8.0 pickleshare 0.7.5 Pillow 8.3.1 Pint 0.17 pip 21.2.3 pip-licenses 3.5.1 plotly 4.14.3 pluggy 0.13.1 ply 3.11 poethepoet 0.9.0 prometheus-client 0.11.0 prompt-toolkit 3.0.19 psycopg2-binary 2.9.1 PTable 0.9.2 ptyprocess 0.7.0 py 1.10.0 pybind11 2.6.2 pybufrkit 0.2.19 pycodestyle 2.7.0 pycparser 2.20 pydantic 1.8.2 pyflakes 2.3.1 Pygments 2.9.0 pyparsing 2.4.7 PyPDF2 1.26.0 pyrsistent 0.18.0 pytest 6.2.4 pytest-cov 2.12.1 pytest-dictsdiff 0.5.8 pytest-notebook 0.6.1 python-dateutil 2.8.2 python-slugify 5.0.2 pythran 0.9.9 pytz 2021.1 PyYAML 5.4.1 pyzmq 22.2.1 qtconsole 5.1.1 QtPy 1.9.0 rapidfuzz 1.4.1 regex 2021.8.3 requests 2.26.0 requests-ftp 0.3.1 requests-unixsocket 0.2.0 retrying 1.3.3 Rx 3.2.0 scipy 1.6.1 selenium 3.141.0 Send2Trash 1.7.1 setuptools 56.1.0 six 1.16.0 smmap 4.0.0 sniffio 1.2.0 snowballstemmer 2.1.0 soupsieve 2.2.1 Sphinx 3.5.4 sphinx-autobuild 2020.9.1 sphinx-autodoc-typehints 1.12.0 sphinx-material 0.0.30 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 sphinxcontrib-svg2pdfconverter 1.1.1 SQLAlchemy 1.3.24 starlette 0.13.6 stevedore 3.3.0 surrogate 0.1 sympy 1.8 tabulate 0.8.9 terminado 0.10.1 testfixtures 6.18.0 testpath 0.5.0 text-unidecode 1.3 toml 0.10.2 tomlkit 0.7.2 toolz 0.11.1 tornado 6.1 tqdm 4.62.0 traitlets 5.0.5 typed-ast 1.4.3 typing-extensions 3.10.0.0 tzlocal 2.1 Unidecode 1.2.0 urllib3 1.26.6 uvicorn 0.13.4 validators 0.18.2 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 1.1.1 Werkzeug 2.0.1 wetterdienst 0.20.3 wheel 0.36.2 widgetsnbextension 3.5.1 wradlib 1.10.3 xarray 0.17.0 xmltodict 0.12.0 zipp 3.5.0

AlexDo1 commented 3 years ago

I just downgraded scipy to version 1.6.1, nothing has changed in the error.

I also executed the sample code in a .py file with the module faulthandler for detailed traceback, this is the output, maybe you can see something in it that could explain the error?

python -Xfaulthandler wetterdienst_test
Fatal Python error: Segmentation fault

Thread 0x000000030a8a1000 (most recent call first):
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/threading.py", line 316 in wait
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/threading.py", line 574 in wait
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/tqdm/_monitor.py", line 60 in run
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/threading.py", line 930 in _bootstrap

Current thread 0x00000002046d9e00 (most recent call first):
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/dogpile/cache/backends/file.py", line 233 in set_serialized
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/dogpile/cache/region.py", line 1287 in _set_cached_value_to_backend
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/dogpile/cache/region.py", line 1012 in gen_value
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/dogpile/lock.py", line 178 in _enter_create
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/dogpile/lock.py", line 94 in _enter
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/dogpile/lock.py", line 185 in __enter__
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/dogpile/cache/region.py", line 1042 in get_or_create
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/dogpile/cache/region.py", line 1577 in get_or_create_for_user_func
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/decorator.py", line 232 in fun
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/wetterdienst/provider/dwd/observation/download.py", line 48 in _download_climate_observations_data
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/wetterdienst/provider/dwd/observation/download.py", line 27 in <listcomp>
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/wetterdienst/provider/dwd/observation/download.py", line 26 in download_climate_observations_data_parallel
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/wetterdienst/provider/dwd/observation/api.py", line 191 in _collect_station_parameter
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/wetterdienst/core/scalar/values.py", line 437 in query
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/tqdm/std.py", line 1185 in __iter__
  File "/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/site-packages/wetterdienst/core/scalar/values.py", line 708 in all
  File "/Users/alexd/Desktop/wetterdienst_test", line 14 in <module>
[1]    25055 segmentation fault  python -Xfaulthandler wetterdienst_test
/Users/alexd/opt/anaconda3/envs/wetterdienst_single/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
gutzbenj commented 3 years ago

Alright, so the error once again seems to be related to the current caching implementation by dogpile.cache. Unfortunately we've seen many problems with this and currently try to use another filecache looking at #431 . At this moment the only fix is to delete the cache. You will find the cache_dir folder running 

wetterdienst show

Delete this folder and try running the code one more time, please.

AlexDo1 commented 3 years ago

I deleted the cache folder and the code now runs without errors.

Thank you so much for helping me (especially on a Sunday noon).

AlexDo1 commented 3 years ago

PS: If at some point you need someone to test the new caching implementation on M1, feel free to contact me.

amotl commented 3 years ago

Hi Alex,

thank you for reporting your observations. Please note, following the rationale by @meteoDaniel at #417, Wetterdienst honors the WD_CACHE_DISABLE environment variable (#426). When set to a truthy value, all caching within Wetterdienst will be turned off completely. While this will raise a significant performance bump, it might save you from such issues recurring.

While I see the good intentions of #489 by @gutzbenj, I believe it is misguided because the culprit for the flaws when concurrently downloading remote resources is not the parallelization on the acquisition side of things, but rather the unsafety of the dbmfile-based cache implementation when accessed concurrently. As such, I believe using WD_CACHE_DISABLE is the quickest way to work around the problem.

If at some point you need someone to test the new caching implementation on M1, feel free to contact me.

On #431 (branch collab/fsspec), the test suite now passes successfully, so I consider it ready to go. Thanks already for willing to take a stab!

With kind regards, Andreas.

AlexDo1 commented 3 years ago

Hello Andreas,

your solution works very well. I only had to delete the wetterdienst cache folder by hand, by setting WD_CACHE_DISABLE=1 after that the data download works smoothly, the performance is absolutely sufficient for my use case.

Thanks again!