search

earthobservations / wetterdienst

Open weather data for humans.
https://wetterdienst.readthedocs.io/
MIT License
349 stars 54 forks source link

_pickle.UnpicklingError: invalid load key, '{'. #233

Closed amotl closed 3 years ago

amotl commented 3 years ago

Describe the bug When cloning the wetterdienst repository, installing its dependencies with Poetry and trying to acquire data using its command line interface, it croaks.

I did this after installing wetterdienst using a different method as outlined within #232.

To reproduce

brew install poetry
git clone https://github.com/earthobservations/wetterdienst
cd wetterdienst
poetry install --extras=influxdb
poetry shell
wetterdienst dwd readings --parameter=air_temperature --resolution=10_minutes --period=recent --station=1048,4411 --date=2020-10/2020-11

Expected behavior Installing and invoking wetterdienst should just work (TM).

Screenshots C'mon ;].

Version information OS: macOS

$ wetterdienst --version
wetterdienst 0.10.1
$ python -V
Python 3.8.6
$ hostinfo
Mach kernel version:
     Darwin Kernel Version 17.7.0: Thu Jun 18 21:21:34 PDT 2020; root:xnu-4570.71.82.5~1/RELEASE_X86_64
Kernel configured for up to 8 processors.
4 processors are physically available.
8 processors are logically available.
Processor type: x86_64h (Intel x86-64h Haswell)
Processors active: 0 1 2 3 4 5 6 7
Primary memory available: 16.00 gigabytes
Default processor set: 344 tasks, 1974 threads, 8 processors
Load average: 1.35, Mach factor: 6.64

Additional context Maybe it's related to the dogpile cache again? See also #217 and #232.

Full traceback

$ wetterdienst dwd readings --parameter=air_temperature --resolution=10_minutes --period=recent --station=1048,4411 --date=2020-10/2020-11
2020-11-20 21:05:47,057 [wetterdienst.dwd.observations.api] INFO   : Acquiring observations data for air_temperature/10_minutes/recent/station_id_1048.
Traceback (most recent call last):
  File "/Users/amo/Library/Caches/pypoetry/virtualenvs/wetterdienst-EkOFQaO8-py3.8/bin/wetterdienst", line 33, in <module>
    sys.exit(load_entry_point('wetterdienst', 'console_scripts', 'wetterdienst')())
  File "/Users/amo/dev/earthobservations/wetterdienst/wetterdienst/cli.py", line 255, in run
    df = observations.collect_safe()
  File "/Users/amo/dev/earthobservations/wetterdienst/wetterdienst/dwd/observations/api.py", line 339, in collect_safe
    data = list(self.collect_data())
  File "/Users/amo/dev/earthobservations/wetterdienst/wetterdienst/dwd/observations/api.py", line 214, in collect_data
    df_parameter = self._collect_data(station_id, parameter_set)
  File "/Users/amo/dev/earthobservations/wetterdienst/wetterdienst/dwd/observations/api.py", line 288, in _collect_data
    df_period = collect_climate_observations_data(
  File "/Users/amo/dev/earthobservations/wetterdienst/wetterdienst/dwd/observations/access.py", line 69, in collect_climate_observations_data
    remote_files = create_file_list_for_climate_observations(
  File "/Users/amo/dev/earthobservations/wetterdienst/wetterdienst/dwd/observations/fileindex.py", line 40, in create_file_list_for_climate_observations
    file_index = create_file_index_for_climate_observations(
  File "<decorator-gen-1>", line 2, in create_file_index_for_climate_observations
  File "/Users/amo/Library/Caches/pypoetry/virtualenvs/wetterdienst-EkOFQaO8-py3.8/lib/python3.8/site-packages/dogpile/cache/region.py", line 1356, in get_or_create_for_user_func
    return self.get_or_create(
  File "/Users/amo/Library/Caches/pypoetry/virtualenvs/wetterdienst-EkOFQaO8-py3.8/lib/python3.8/site-packages/dogpile/cache/region.py", line 954, in get_or_create
    with Lock(
  File "/Users/amo/Library/Caches/pypoetry/virtualenvs/wetterdienst-EkOFQaO8-py3.8/lib/python3.8/site-packages/dogpile/lock.py", line 185, in __enter__
    return self._enter()
  File "/Users/amo/Library/Caches/pypoetry/virtualenvs/wetterdienst-EkOFQaO8-py3.8/lib/python3.8/site-packages/dogpile/lock.py", line 87, in _enter
    value = value_fn()
  File "/Users/amo/Library/Caches/pypoetry/virtualenvs/wetterdienst-EkOFQaO8-py3.8/lib/python3.8/site-packages/dogpile/cache/region.py", line 899, in get_value
    value = self.backend.get(key)
  File "/Users/amo/Library/Caches/pypoetry/virtualenvs/wetterdienst-EkOFQaO8-py3.8/lib/python3.8/site-packages/dogpile/cache/backends/file.py", line 231, in get
    value = pickle.loads(value)
_pickle.UnpicklingError: invalid load key, '{'.
amotl commented 3 years ago

When purging the cache using

rm -r ~/Library/Caches/wetterdienst

it started working again.

It is obvious that we see some Pickle version issue here. After switching back to the virtualenv-based installation (#232), I see the same error again which is outlined there (UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte).

I wonder why this happens as we are using Python 3.8.6 in both cases. I can confirm it is even the very same interpreter, installed using pyenv.

Python 3.8.6 (default, Oct 27 2020, 08:58:03)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.