measurements.to_xarray() is possible, but this will give you a huge dataset and that is a bit inconvenient. ddlpy.simplify_dataframe(measurements).to_xarray() is better, but it will still give you all codes/omschrijvingen separately. For BAALHK 1980-2024 this gives a dataset of >1GB mainly thanks to the WaardeBepalingsmethode.Omschrijving variable.
[x] It would be great to set these as an attribute table to WaardeBepalingsmethode.Code, but showing all available values (https://github.com/Deltares/ddlpy/issues/28). >> no
[x] Also consider combining ds.attrs by merging Code/Omschrijving attrs as f"{Code} ({Omschrijving})". >> no add list of code/omschrijving keys/values to variables
[x] Furthermore, consider to automatically upscale strings containing only numeric values to integers. This would probably reduce WaardeBepalingsmethode.Code significantly. It might be wise to do this already earlier in the process, although this might break some pipelines. >> not generic
[x] consider to always keep some columns as variables (like kwaliteitswaardecode, status, maybe others), since that would make concatenating datasets way easier.
Example code that retrieves VLISSGN data with multiple Waardebepalingsmethode (changed on 07-09-1993):
time
1990-02-27 01:00:00+01:00 other:F039
1990-03-01 00:00:00+01:00 other:F027
Name: WaardeBepalingsmethode.Code, dtype: object
full [MB]: 1.801
simple [MB]: 0.267
Update 27-3-2023: With the catalog_filter argument added to ddlpy.locations() or the private ddlpy.ddlpy.catalog() function added in https://github.com/Deltares/ddlpy/pull/87, retrieving the extended catalog is easy. We can also subset the catalog dataframe directly:
import pandas as pd
import ddlpy
catalog_filter = ['Compartimenten','Eenheden','Grootheden',
'Hoedanigheden','Groeperingen','MeetApparaten',
'Typeringen','WaardeBepalingsmethoden','WaardeBewerkingsmethoden','Parameters']
if 1:
result = ddlpy.ddlpy.catalog(catalog_filter=catalog_filter)
cat_aquometadatalijst = pd.json_normalize(result["AquoMetadataLijst"])
else:
locations = ddlpy.locations(catalog_filter=catalog_filter)
cat_aquometadatalijst = locations
# bool_grootheid = locations["Grootheid.Code"].isin(["WATHTE"])
# bool_groepering = locations["Groepering.Code"].isin(["NVT"])
# selected = locations.loc[bool_grootheid & bool_groepering]
# cat_aquometadatalijst = selected
key_list_full = [x.replace(".Code","") for x in cat_aquometadatalijst.columns if x.endswith(".Code")]
# >> ['Compartiment', 'Eenheid', 'Grootheid', 'Hoedanigheid', 'MeetApparaat', 'Parameter', 'Typering', 'Groepering', 'WaardeBepalingsmethode']
for key in key_list_full:
df_meta_unique = cat_aquometadatalijst[[f"{key}.Code",f"{key}.Omschrijving"]].drop_duplicates()
print(f'\n{len(df_meta_unique)} unique {key} available in requested subset')
print(df_meta_unique)
dict_meta_unique = df_meta_unique.set_index(f"{key}.Code")[f"{key}.Omschrijving"].to_dict()
This can be used for adding metadata attrs, but the full set of Waardebepalingsmethode and Parameter are way to large to add:
7 unique Compartiment available in requested subset
51 unique Eenheid available in requested subset
104 unique Grootheid available in requested subset
71 unique Hoedanigheid available in requested subset
34 unique MeetApparaat available in requested subset
895 unique Parameter available in requested subset
6 unique Typering available in requested subset
5 unique Groepering available in requested subset
510 unique WaardeBepalingsmethode available in requested subset
6 unique WaardeBewerkingsmethode available in requested subset
Instead, the unique values from the subsetted extended locations dataframe can be used. This is better than using the unique values from the measurements dataframe, since these will differ per station and time range.
measurements.to_xarray()is possible, but this will give you a huge dataset and that is a bit inconvenient.ddlpy.simplify_dataframe(measurements).to_xarray()is better, but it will still give you all codes/omschrijvingen separately. For BAALHK 1980-2024 this gives a dataset of >1GB mainly thanks to theWaardeBepalingsmethode.Omschrijvingvariable.Todo:
WaardeBepalingsmethode.Code, but showing all available values (https://github.com/Deltares/ddlpy/issues/28). >> nods.attrsby merging Code/Omschrijving attrs asf"{Code} ({Omschrijving})". >> no add list of code/omschrijving keys/values to variablesWaardeBepalingsmethode.Codesignificantly. It might be wise to do this already earlier in the process, although this might break some pipelines. >> not genericExample code that retrieves VLISSGN data with multiple Waardebepalingsmethode (changed on 07-09-1993):
Alternative example (already works):
Prints:
Update 27-3-2023: With the
catalog_filterargument added toddlpy.locations()or the privateddlpy.ddlpy.catalog()function added in https://github.com/Deltares/ddlpy/pull/87, retrieving the extended catalog is easy. We can also subset the catalog dataframe directly:This can be used for adding metadata attrs, but the full set of Waardebepalingsmethode and Parameter are way to large to add:
Instead, the unique values from the subsetted extended locations dataframe can be used. This is better than using the unique values from the measurements dataframe, since these will differ per station and time range.