amotl
commented
3 years ago Dear Hendrik,
thank you for writing in.
We are also currently unhappy with the current cache implementation. #431 will improve it.
Currently, we are seeing issues with parallel operation of Wetterdienst, because the .dbm-based implementation is apparently not thread safe in all scenarios.
So, currently you have two options: Either you can try running Wetterdienst in a sandbox environment, incorporating the patch from #431, or you can turn off caching altogether by using the respective environment variable. While I currently can't remember it, Benjamin will sure be happy to tell you about it.
After I am back from vacation around July, 15th, I will take care of integrating #431 in order to improve the situation of operating Wetterdienst in multi-threaded environments.
Cheers, Andreas.
On 7 July 2021 13:45:27 CEST, HendrikHuel @.***> wrote:
Describe the bug> I am trying to download data for a range of stations from DWD using the simple code provided below. During the process I noticed some pagefile errors and investigated. It seems that the payloadfile blows up in size. While looping over my list of stations the code doubles the size of _payload5m.dbm.db. >
I assume I has something todo with the parrallel downloading, but I was unable to confirm. >
Is it possible to limit the number of threads to one or to turn of parrallel downloading completely? >
To Reproduce> Here is my code.>
from datetime import datetime> from pathlib import Path> from tqdm import tqdm> import pandas as pd> from wetterdienst.provider.dwd.observation import (> DwdObservationParameter,> DwdObservationRequest,> DwdObservationPeriod,> DwdObservationResolution,> )> # path to payload file> full_path_to_payload_file = (> "/Users/hhuelsbusch/Library/Caches/wetterdienst/payload_5m.dbm.db"> )> # Statsion for which data should be collected> list_stations = ['13713', '05404', '04154', '02928', '03137', '00232', '03659',> '03167', '04018', '03390', '03591', '01300', '01420', '02578',> '00399', '02947', '04371', '04411', '05064', '03379', '02483',> '03385', '02932', '02532', '02968', '03540', '04275', '03028',> '00603', '07431', '02522', '07367', '00150', '04625', '03382',> '00390', '03404', '02691', '00598', '02211', '00919', '04926',> '04177', '01766', '03815', '03490', '03623', '04928', '01515',> '03362', '03098', '00161', '02497', '04349', '01424', '03031',> '03246', '04560', '00403', '07341', '01072', '00142', '05300',> '01421', '01262', '05717', '07106', '00424', '00433', '07330',> '01327', '04301', '00348', '01863', '06346', '03575', '00691',> '01691', '04692', '03340', '02656', '04997', '04323', '00599',> '04160', '00596', '04584', '01425', '01964', '01590', '02601',> '00326', '05279', '00257', '00430', '04508', '04651', '01246',> '02110', '01078', '04127', '05541', '03925', '13776', '00766',> '15000', '03660', '06337', '03139', '05426', '04927', '05014',> '00917', '00003', '03155', '02480', '02667', '01103', '02362',> '06264', '05480']> df_wd = []> list_failed_stations = []> for s in tqdm(list_stations):> print(f"Station: {s}")> request = DwdObservationRequest(> parameter=[> DwdObservationParameter.DAILY.TEMPERATURE_AIR_200,> DwdObservationParameter.DAILY.TEMPERATURE_AIR_MAX_200,> DwdObservationParameter.DAILY.TEMPERATURE_AIR_MIN_200,> DwdObservationParameter.DAILY.PRECIPITATION_HEIGHT,> DwdObservationParameter.DAILY.SNOW_DEPTH,> DwdObservationParameter.DAILY.WIND_SPEED,> ],> resolution=DwdObservationResolution.DAILY,> period=DwdObservationPeriod.HISTORICAL,> start_date=datetime(2017, 1, 1),> end_date=datetime(2020, 12, 31),> tidy=True,> humanize=True,> ).filter_by_station_id(> station_id=[int(s)]> ) # Request data for just one station. Setting 'station_id=list_station' results in an other error.> df_sd = request.values.all().df> size = (> Path(full_path_to_payload_file).stat().st_size / 1024 / 1024> ) # Calculate size of payload file> print(f"Size of payloadfile: {size:0.2f}MB")> df_wd.append(df_sd)> df_wd = pd.concat(df_wd)> ```> And the print from my ipython console, which shows the doubling of size at some steps.> ```> runfile('/Users/hhuelsbusch/RAM_Projects/aktionsplanung/src/EDA/bug_in_wetterdienst.py')> 0%| | 0/121 [00:00<?, ?it/s]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Station: 13713> 1%| | 1/121 [00:02<04:50, 2.42s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 0.63MB> Station: 05404> 2%|▏ | 2/121 [00:11<12:57, 6.53s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 4.56MB> Station: 04154> 2%|▏ | 3/121 [00:12<07:57, 4.05s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 4.91MB> Station: 02928> 3%|▎ | 4/121 [00:29<17:27, 8.95s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 11.53MB> Station: 03137> 4%|▍ | 5/121 [00:37<17:00, 8.80s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 14.96MB> Station: 00232> 5%|▍ | 6/121 [00:46<17:00, 8.88s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 18.63MB> Station: 03659> 6%|▌ | 7/121 [00:52<15:02, 7.92s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 20.94MB> Station: 03167> 7%|▋ | 8/121 [00:59<14:21, 7.62s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 23.90MB> Station: 04018> 7%|▋ | 9/121 [01:03<12:06, 6.49s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 25.48MB> Station: 03390> 8%|▊ | 10/121 [01:09<11:45, 6.36s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 27.92MB> Station: 03591> 9%|▉ | 11/121 [01:13<09:58, 5.44s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 29.32MB> Station: 01300> 10%|▉ | 12/121 [01:22<11:55, 6.56s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 32.89MB> Station: 01420> 11%|█ | 13/121 [01:31<12:58, 7.21s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 36.45MB> Station: 02578> 12%|█▏ | 14/121 [01:47<17:36, 9.87s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 43.15MB> Station: 00399> 12%|█▏ | 15/121 [01:51<14:36, 8.27s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 44.84MB> Station: 02947> 13%|█▎ | 16/121 [01:58<13:46, 7.88s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 48.11MB> Station: 04371> 14%|█▍ | 17/121 [02:08<14:33, 8.40s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 52.10MB> Station: 04411> 15%|█▍ | 18/121 [02:13<12:39, 7.37s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 55.07MB> Station: 05064> 16%|█▌ | 19/121 [02:15<09:47, 5.76s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 55.83MB> Station: 03379> 17%|█▋ | 20/121 [02:23<10:48, 6.42s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 59.02MB> Station: 02483> 17%|█▋ | 21/121 [02:33<12:29, 7.50s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 65.13MB> Station: 03385> 18%|█▊ | 22/121 [02:38<11:18, 6.85s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 67.27MB> Station: 02932> 19%|█▉ | 23/121 [02:46<11:37, 7.11s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 70.45MB> Station: 02532> 20%|█▉ | 24/121 [02:54<11:50, 7.33s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 77.59MB> Station: 02968> 21%|██ | 25/121 [03:02<12:26, 7.78s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 81.16MB> Station: 03540> 21%|██▏ | 26/121 [03:06<10:16, 6.49s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 82.50MB> Station: 04275> 22%|██▏ | 27/121 [03:14<11:05, 7.08s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 93.82MB> Station: 03028> 23%|██▎ | 28/121 [03:20<10:30, 6.78s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 96.31MB> Station: 00603> 24%|██▍ | 29/121 [03:25<09:09, 5.97s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 97.86MB> Station: 07431> 25%|██▍ | 30/121 [03:26<07:07, 4.69s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 98.47MB> Station: 02522> 26%|██▌ | 31/121 [03:42<11:53, 7.92s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 120.76MB> Station: 07367> 26%|██▋ | 32/121 [03:43<08:57, 6.04s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 121.42MB> Station: 00150> 27%|██▋ | 33/121 [03:52<09:46, 6.66s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 124.74MB> Station: 04625> 28%|██▊ | 34/121 [04:06<13:10, 9.08s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 162.79MB> Station: 03382> 29%|██▉ | 35/121 [04:07<09:33, 6.67s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 163.18MB> Station: 00390> 30%|██▉ | 36/121 [04:11<08:23, 5.93s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 164.80MB> Station: 03404> 31%|███ | 37/121 [04:23<10:42, 7.65s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 233.47MB> Station: 02691> 31%|███▏ | 38/121 [04:25<08:11, 5.92s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 234.18MB> Station: 00598> 32%|███▏ | 39/121 [04:26<06:10, 4.52s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 234.61MB> Station: 02211> 33%|███▎ | 40/121 [04:36<08:11, 6.06s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 366.39MB> Station: 00919> 34%|███▍ | 41/121 [04:40<07:25, 5.57s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 368.13MB> Station: 04926> 35%|███▍ | 42/121 [04:45<06:56, 5.27s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 369.91MB> Station: 04177> 36%|███▌ | 43/121 [04:51<07:15, 5.58s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 372.33MB> Station: 01766> 36%|███▋ | 44/121 [04:55<06:28, 5.05s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 629.89MB> Station: 03815> 37%|███▋ | 45/121 [05:02<07:11, 5.68s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 632.83MB> Station: 03490> 38%|███▊ | 46/121 [05:11<08:09, 6.53s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 636.32MB> Station: 03623> 39%|███▉ | 47/121 [05:17<07:48, 6.34s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 1150.50MB> Station: 04928> 40%|███▉ | 48/121 [05:24<08:10, 6.72s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 1153.59MB> Station: 01515> 40%|████ | 49/121 [05:27<06:36, 5.51s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 1154.46MB> Station: 03362> 41%|████▏ | 50/121 [05:30<05:35, 4.73s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 1155.59MB> Station: 03098> 42%|████▏ | 51/121 [05:33<05:03, 4.34s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 2180.94MB> Station: 00161> 43%|████▎ | 52/121 [05:37<04:46, 4.15s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 2182.25MB> Station: 02497> 44%|████▍ | 53/121 [05:46<06:19, 5.59s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 2185.78MB> Station: 04349> 45%|████▍ | 54/121 [05:50<05:50, 5.23s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 2187.32MB> Station: 01424> 45%|████▌ | 55/121 [05:55<05:25, 4.94s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 4236.95MB> Station: 03031> 46%|████▋ | 56/121 [05:59<05:20, 4.94s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 4238.89MB> Station: 03246> 47%|████▋ | 57/121 [06:02<04:33, 4.28s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 4239.82MB> Station: 04560> 48%|████▊ | 58/121 [06:11<05:59, 5.71s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 4243.26MB> Station: 00403> 49%|████▉ | 59/121 [06:20<06:48, 6.59s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 8342.69MB> Station: 07341> 50%|████▉ | 60/121 [06:22<05:19, 5.24s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 8343.46MB> Station: 01072> 50%|█████ | 61/121 [06:30<06:06, 6.11s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 8346.58MB> Station: 00142> 51%|█████ | 62/121 [06:38<06:35, 6.70s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 16541.65MB> Station: 05300> 52%|█████▏ | 63/121 [06:46<06:49, 7.06s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 16544.70MB> Station: 01421> 53%|█████▎ | 64/121 [06:49<05:31, 5.82s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 16545.78MB> Station: 01262> 54%|█████▎ | 65/121 [06:53<04:52, 5.22s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 16547.21MB> Station: 05717> 55%|█████▍ | 66/121 [07:02<05:58, 6.53s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 32934.88MB> Station: 07106> 55%|█████▌ | 67/121 [07:04<04:37, 5.14s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 32935.57MB> Station: 00424> 56%|█████▌ | 68/121 [07:08<04:04, 4.62s/it]start_date and end_date filtering limited to defined periods [<Period.HISTORICAL: 'historical'>]> Size of payloadfile: 32936.83MB> ```> **Screenshots**> If applicable, add screenshots to help explain your problem.> **Desktop (please complete the following information):**> - OS: MAC OS> - My python environment> ```> # Name Version Build Channel> aenum 3.1.0 pypi_0 pypi> anyio 3.2.1 py37hf985489_0 conda-forge> appdirs 1.4.4 pypi_0 pypi> appnope 0.1.2 py37hf985489_1 conda-forge> argon2-cffi 20.1.0 py37hf967b71_2 conda-forge> async_generator 1.10 py_0 conda-forge> attrs 21.2.0 pyhd8ed1ab_0 conda-forge> babel 2.9.1 pyh44b312d_0 conda-forge> backcall 0.2.0 pyh9f0ad1d_0 conda-forge> backports 1.0 py_2 conda-forge> backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge> beautifulsoup4 4.9.3 pypi_0 pypi> blas 1.0 mkl > bleach 3.3.0 pyh44b312d_0 conda-forge> brotlipy 0.7.0 py37h271585c_1001 conda-forge> ca-certificates 2021.7.5 hecd8cb5_1 > cachetools 4.2.2 pypi_0 pypi> certifi 2021.5.30 py37hecd8cb5_0 > cffi 1.14.5 py37hda351d9_0 conda-forge> chardet 4.0.0 py37hf985489_1 conda-forge> click 7.1.2 pypi_0 pypi> click-params 0.1.2 pypi_0 pypi> cloudpickle 1.6.0 py_0 > cloup 0.8.2 pypi_0 pypi> cryptography 3.4.7 py37hce4a858_0 conda-forge> cycler 0.10.0 py37_0 > dateparser 1.0.0 pypi_0 pypi> decorator 4.4.2 pyhd3eb1b0_0 > defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge> deprecation 2.1.0 pypi_0 pypi> dogpile-cache 1.1.3 pypi_0 pypi> entrypoints 0.3 pyhd8ed1ab_1003 conda-forge> freetype 2.10.4 ha233b18_0 > idna 2.10 pyh9f0ad1d_0 conda-forge> importlib-metadata 1.7.0 pypi_0 pypi> intel-openmp 2021.2.0 hecd8cb5_564 > ipykernel 5.5.5 py37h85f7c60_0 conda-forge> ipython 7.25.0 py37h85f7c60_1 conda-forge> ipython_genutils 0.2.0 py_1 conda-forge> jedi 0.18.0 py37hf985489_2 conda-forge> jinja2 3.0.1 pyhd8ed1ab_0 conda-forge> jpeg 9b he5867d9_2 > json5 0.9.5 pyh9f0ad1d_0 conda-forge> jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge> jupyter_client 6.1.12 pyhd8ed1ab_0 conda-forge> jupyter_core 4.7.1 py37hf985489_0 conda-forge> jupyter_server 1.9.0 pyhd8ed1ab_0 conda-forge> jupyterlab 3.0.16 pyhd8ed1ab_0 conda-forge> jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge> jupyterlab_server 2.6.0 pyhd8ed1ab_0 conda-forge> kiwisolver 1.3.1 py37h23ab428_0 > lcms2 2.12 hf1fd2bf_0 > libcxx 10.0.0 1 > libffi 3.3 hb1e8313_2 > libpng 1.6.37 ha441bb4_0 > libsodium 1.0.18 hbcb3906_1 conda-forge> libtiff 4.2.0 h87d7836_0 > libwebp-base 1.2.0 h9ed2024_0 > lxml 4.6.3 pypi_0 pypi> lz4-c 1.9.3 h23ab428_0 > markupsafe 2.0.1 py37h271585c_0 conda-forge> matplotlib 3.3.4 py37hecd8cb5_0 > matplotlib-base 3.3.4 py37h8b3ea08_0 > matplotlib-inline 0.1.2 pyhd8ed1ab_2 conda-forge> measurement 3.2.0 pypi_0 pypi> mistune 0.8.4 py37h271585c_1004 conda-forge> mkl 2021.2.0 hecd8cb5_269 > mkl-service 2.3.0 py37h9ed2024_1 > mkl_fft 1.3.0 py37h4a7008c_2 > mkl_random 1.2.1 py37hb2f4e1b_2 > mpmath 1.2.1 pypi_0 pypi> nbclassic 0.3.1 pyhd8ed1ab_1 conda-forge> nbclient 0.5.3 pyhd8ed1ab_0 conda-forge> nbconvert 6.1.0 py37hf985489_0 conda-forge> nbformat 5.1.3 pyhd8ed1ab_0 conda-forge> ncurses 6.2 h0a44026_1 > nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge> notebook 6.4.0 pyha770c72_0 conda-forge> numpy 1.20.2 py37h4b4dc7a_0 > numpy-base 1.20.2 py37he0bd621_0 > olefile 0.46 py37_0 > openssl 1.1.1k h9ed2024_0 > packaging 21.0 pyhd8ed1ab_0 conda-forge> pandas 1.2.5 py37h23ab428_0 > pandoc 2.14.0.3 h0d85af4_0 conda-forge> pandocfilters 1.4.2 py_1 conda-forge> parso 0.8.2 pyhd8ed1ab_0 conda-forge> pbr 5.6.0 pypi_0 pypi> pexpect 4.8.0 pyh9f0ad1d_2 conda-forge> pickleshare 0.7.5 py_1003 conda-forge> pillow 8.2.0 py37h5270095_0 > pint 0.17 pypi_0 pypi> pip 21.1.3 py37hecd8cb5_0 > prometheus_client 0.11.0 pyhd8ed1ab_0 conda-forge> prompt-toolkit 3.0.19 pyha770c72_0 conda-forge> ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge> pycparser 2.20 pyh9f0ad1d_2 conda-forge> pygments 2.9.0 pyhd8ed1ab_0 conda-forge> pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge> pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge> pypdf2 1.26.0 pypi_0 pypi> pyrsistent 0.17.3 py37h271585c_2 conda-forge> pysocks 1.7.1 py37hf985489_3 conda-forge> python 3.7.10 h88f2d9e_0 > python-dateutil 2.8.1 py_0 conda-forge> python_abi 3.7 2_cp37m conda-forge> pytz 2021.1 pyhd8ed1ab_0 conda-forge> pyzmq 20.0.0 py37h23ab428_1 > rapidfuzz 1.4.1 pypi_0 pypi> readline 8.1 h9ed2024_0 > regex 2021.7.6 pypi_0 pypi> requests 2.25.1 pyhd3deb0d_0 conda-forge> requests-ftp 0.3.1 pypi_0 pypi> requests-unixsocket 0.2.0 py_0 conda-forge> scipy 1.7.0 pypi_0 pypi> send2trash 1.7.1 pyhd8ed1ab_0 conda-forge> setuptools 52.0.0 py37hecd8cb5_0 > six 1.16.0 pyh6c4a22f_0 conda-forge> sniffio 1.2.0 py37hf985489_1 conda-forge> soupsieve 2.2.1 pypi_0 pypi> spyder-kernels 2.0.3 py37hecd8cb5_0 > sqlite 3.36.0 hce871da_0 > stevedore 3.3.0 pypi_0 pypi> sympy 1.8 pypi_0 pypi> tabulate 0.8.9 pypi_0 pypi> terminado 0.10.1 py37hf985489_0 conda-forge> testpath 0.5.0 pyhd8ed1ab_0 conda-forge> tk 8.6.10 hb0a8c7a_0 > tornado 6.1 py37h271585c_1 conda-forge> tqdm 4.61.2 pypi_0 pypi> traitlets 5.0.5 py_0 conda-forge> typing_extensions 3.10.0.0 pyha770c72_0 conda-forge> tzlocal 2.1 pypi_0 pypi> urllib3 1.26.6 pyhd8ed1ab_0 conda-forge> validators 0.18.2 pypi_0 pypi> wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge> webencodings 0.5.1 py_1 conda-forge> websocket-client 0.57.0 py37hf985489_4 conda-forge> wetterdienst 0.20.2 pypi_0 pypi> wheel 0.36.2 pyhd3eb1b0_0 > wurlitzer 2.1.0 py37hecd8cb5_0 > xz 5.2.5 h1de35cc_0 > zeromq 4.3.4 h23ab428_0 > zipp 3.5.0 pyhd8ed1ab_0 conda-forge> zlib 1.2.11 h1de35cc_3 > zstd 1.4.9 h322a384_0 > ```> Please let me know if more informations are needed, or if I can turn on/off some stuff to help to identify the problem.> -- > You are receiving this because you are subscribed to this thread.> Reply to this email directly or view it on GitHub:> https://github.com/earthobservations/wetterdienst/issues/474
-- Sent from my mind. This might have been typed on a mobile device, so please excuse my brevity.
HendrikHuel
gutzbenj
Describe the bug I am trying to download data for a range of stations from DWD using the simple code provided below. During the process I noticed some pagefile errors and investigated. It seems that the payloadfile blows up in size. While looping over my list of stations the code doubles the size of _payload5m.dbm.db.
I assume I has something todo with the parrallel downloading, but I was unable to confirm.
Is it possible to limit the number of threads to one or to turn of parrallel downloading completely?
To Reproduce Here is my code.
And the print from my ipython console, which shows the doubling of size at some steps.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Please let me know if more informations are needed, or if I can turn on/off some stuff to help to identify the problem.