search

earthobservations / wetterdienst

Open weather data for humans.
https://wetterdienst.readthedocs.io/
MIT License
340 stars 52 forks source link

DwdObservationRequest and dataframe from values #1316

Open oschloem opened 1 week ago

oschloem commented 1 week ago

Describe the bug When running the _DwdObservationRequest command_, I get a value error 'ValueError: not enough values to unpack (expected 2, got 1)'. At this the command: 'values = request.values.all().df' is crucial. I know that there have been some problems with the data request in the past because the DWD has changed its data structure. Is this still the case? Many thanks in advance for your reply and help!

To Reproduce cf: https://pypi.org/project/wetterdienst/

import polars as pl _ = pl.Config.set_tbl_hide_dataframe_shape(True) from wetterdienst import Settings from wetterdienst.provider.dwd.observation import DwdObservationRequest settings = Settings( # default ... ts_shape="long", # tidy data ... ts_humanize=True, # humanized parameters ... ts_si_units=True # convert values to SI units ... ) request = DwdObservationRequest( ... parameter="climate_summary", ... resolution="daily", ... start_date="1990-01-01", # if not given timezone defaulted to UTC ... end_date="2020-01-01", # if not given timezone defaulted to UTC ... settings=settings ... ).filter_by_station_id(station_id=(1048, 4411)) stations = request.df stations.head()

values = request.values.all().df values.head()

Expected behavior The values should be in a polars dataframe

Desktop (please complete the following information):

vidyaranya92 commented 1 week ago

having similar issue, same code with previous version seems to be working fine.

gutzbenj commented 1 week ago

Hi @oschloem @vidyaranya92 ,

thanks for reporting! Will have a look later!

steffenrunge commented 6 days ago

I have the same issues suddenly on any of my own scripts (that worked perfectly for many weeks) as well as on any proven templates.

The error is always:

ValueError Traceback (most recent call last)

in <cell line: 19>() 17 stations = request.df 18 stations.head() ---> 19 values = request.values.all().df 20 values.head()

ValueError: not enough values to unpack (expected 2, got 1)

Example script

import polars as pl
_ = pl.Config.set_tbl_hide_dataframe_shape(True)
from wetterdienst import Settings
from wetterdienst.provider.dwd.observation import DwdObservationRequest
settings = Settings( # default
ts_shape="long",  # tidy data
ts_humanize=True,  # humanized parameters
ts_si_units=True  # convert values to SI units
)
request = DwdObservationRequest(
parameter="climate_summary",
resolution="daily",
start_date="1990-01-01",  # if not given timezone defaulted to UTC
end_date="2020-01-01",  # if not given timezone defaulted to UTC
settings=settings
).filter_by_station_id(station_id=(1048, 4411))
stations = request.df
stations.head()
values = request.values.all().df
values.head()
gutzbenj commented 6 days ago

polars group_by signature has changed in version 1.0.0. I'll hand in a fix asap.

gutzbenj commented 6 days ago

Just release a new version, that should fix it. Please give it a try!

j0nes2k commented 6 days ago

@gutzbenj Thank you for your work on this! However I still get a different error. Here is my code and exception:

parameters_10min = [
    "wind_speed",
    "wind_direction",
    "wind_gust_max",
    "temperature_air_mean_200",
    "pressure_air_site",
]

    start_time = datetime.now(timezone.utc) - timedelta(minutes=60)
    data_10min = DwdObservationRequest(
        parameter=parameters_10min,
        resolution=Resolution.MINUTE_10,
        period=Period.NOW,
        start_date=start_time,
        end_date=datetime.now(timezone.utc),
    )

    data_10min_df = data_10min.all().values.all().df

...which results in the following exception:

    data_10min_df = data_10min.all().values.all().df  
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/virtualenvs/weatherdata/lib/python3.11/site-packages/wetterdienst/core/timeseries/values.py", line 574, in all
    for result in tqdm(self.query(), total=len(self.sr.station_id), file=tqdm_out):
  File "/workspace/virtualenvs/weatherdata/lib/python3.11/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/workspace/virtualenvs/weatherdata/lib/python3.11/site-packages/wetterdienst/core/timeseries/values.py", line 428, in query
    parameter_df = self._collect_station_parameter(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/virtualenvs/weatherdata/lib/python3.11/site-packages/wetterdienst/provider/dwd/observation/api.py", line 169, in _collect_station_parameter
    parameter_df = parameter_df.collect()
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/virtualenvs/weatherdata/lib/python3.11/site-packages/polars/lazyframe/frame.py", line 1942, in collect
    return wrap_df(ldf.collect(callback))
                   ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ColumnNotFoundError: mess_datum

This error occurred with the following context stack:
        [1] 'with_columns' failed
        [2] 'select' input failed to resolve
        [3] 'unique' input failed to resolve

Environment:

Please let me know if this is related or if you need more information!

steffenrunge commented 6 days ago

The issue is RESOLVED ✅

@gutzbenj Thank you very much for the quick resolution!!👍 🥇

j0nes2k commented 5 days ago

Just a short update: this error does not occur anymore since several hours using the same setup as above, therefore I cannot reproduce this any more. What I found in my downloads however is a data file that I downloaded during the time where this error occurs. The filename is 10minutenwerte_TU_00259_now.zip, Here are some lines from this file:

STATIONS_ID                      ;MESS_DATUM                -999;QN-999;PP_10      -999;TT_10      -999;TM5_10        -999;RF_10      -999;TD_10;eor
        259;202407010000                    -999;    2      -999;                 -999;  16.1        -999;  14.1        -999;              -999;-999;eor
        259;202407010010                    -999;    2      -999;                 -999;  15.8        -999;  14.2        -999;              -999;-999;eor
        259;202407010020                    -999;    2      -999;                 -999;  15.8        -999;  13.8        -999;              -999;-999;eor
        259;202407010030                    -999;    2      -999;                 -999;  15.9        -999;  13.8        -999;              -999;-999;eor
        259;202407010040                    -999;    2      -999;                 -999;  16.0        -999;  13.9        -999;              -999;-999;eor
        259;202407010050                    -999;    2      -999;                 -999;  15.9        -999;  13.8        -999;              -999;-999;eor
        259;202407010100                    -999;    2      -999;                 -999;  15.6        -999;  13.3        -999;              -999;-999;eor
        259;202407010110                    -999;    2      -999;                 -999;  15.4        -999;  12.8        -999;              -999;-999;eor
        259;202407010120                    -999;    2      -999;                 -999;  15.1        -999;  12.5        -999;              -999;-999;eor
        259;202407010130                    -999;    2      -999;                 -999;  14.6        -999;  12.3        -999;  96.3        -999;  14.0;eor
        259;202407010140                    -999;    2      -999;                 -999;  14.4        -999;  12.3        -999;  97.2        -999;  14.0;eor
        259;202407010150                    -999;    2      -999;                 -999;  14.6        -999;  12.2        -999;  97.5        -999;  14.2;eor
        259;202407010200                    -999;    2      -999;                 -999;  14.7        -999;  12.3        -999;  96.3        -999;  14.1;eor
        259;202407010210                    -999;    2      -999;                 -999;  14.5        -999;  12.4        -999;  96.3        -999;  13.9;eor
        259;202407010220                    -999;    2      -999;                 -999;  14.7        -999;  12.4        -999;  96.7        -999;  14.2;eor
        259;202407010230                    -999;    2      -999;                 -999;  14.6        -999;  12.0        -999;  96.9        -999;  14.1;eor
        259;202407010240                    -999;    2      -999;                 -999;  13.8        -999;  11.7        -999;  96.8        -999;  13.3;eor
        259;202407010250                    -999;    2      -999;                 -999;  13.8        -999;  11.3        -999;  98.5        -999;  13.6;eor
        259;202407010300                    -999;    2      -999;                 -999;  14.5        -999;  11.4        -999;  95.5        -999;  13.8;eor
        259;202407010310                    -999;    2      -999;                 -999;  15.0        -999;  11.8        -999;  93.6        -999;  14.0;eor
        259;202407010320                    -999;    2      -999;                 -999;  15.0        -999;  12.1        -999;  92.6        -999;  13.8;eor
        259;202407010330                    -999;    2      -999;                 -999;  15.2        -999;  12.0        -999;  93.1        -999;  14.1;eor
        259;202407010340                    -999;    2      -999;                 -999;  14.9        -999;  12.9        -999;  92.7        -999;  13.7;eor
        259;202407010350                    -999;    2      -999;                 -999;  15.4        -999;  12.2        -999;  90.7        -999;  13.9;eor
        259;202407010400                    -999;    2      -999;                 -999;  15.3        -999;  12.4        -999;  90.6        -999;  13.8;eor
        259;202407010410                    -999;    2      -999;                 -999;  15.4        -999;  13.3        -999;  89.9        -999;  13.8;eor
        259;202407010420                    -999;    2      -999;                 -999;  15.4        -999;  13.7        -999;  90.0        -999;  13.8;eor
        259;202407010430                    -999;    2      -999;                 -999;  15.8        -999;  14.2        -999;  88.3        -999;  13.9;eor
        259;202407010440                    -999;    2      -999;                 -999;  15.8        -999;  13.9        -999;  87.7        -999;  13.8;eor
        259;202407010450                    -999;    2      -999;                 -999;  15.7        -999;  13.8        -999;  87.7        -999;  13.7;eor
        259;202407010500                    -999;    2      -999;                 -999;  16.0        -999;  13.8        -999;  86.1        -999;  13.7;eor

Maybe this still helps...

oschloem commented 5 days ago

Thanks @gutzbenj! 0.89 works perfect!

gutzbenj commented 5 days ago

@gutzbenj Thank you for your work on this! However I still get a different error. Here is my code and exception:

I dont get any error with this code, it just takes a lot of time due to high resolution.

j0nes2k commented 5 days ago

@gutzbenj We do not see this error anymore either. This happened for approx. 2 days and was reproducible. I will let you know when this occurs again in a separate issue.

gutzbenj commented 5 days ago

@j0nes2k Thanks for reporting and glad that it works again! It could be that the file was corrupt during this time. We've already had this error once with one of the station lists of DWD where the file just had an error message about the file writing process in it.