LTER-LIFE / FDFDT

FAIR Data for Digital Twins
0 stars 0 forks source link

Retrieve data from KNMI through Open API #6

Closed CherineJ closed 5 months ago

CherineJ commented 5 months ago

As stated here in more detail, the KNMI Open API allows to retrieve single netCDF files per day. To retrieve not only the first or last 10 files of a data set, it is necessary to know the exact file names, so that for each file a separate temporary download URL can be requested and the file downloaded. The naming of the files follows the general pattern:

INTER_OPER_R_[Data set abbreviation in CAPS]___L3[startday as YYYYMMDD]T000000_[end day as YYYYMMDD]T000000_0006.nc e.g.: "INTER_OPER_RTG1L3__20240220T000000_20240221T000000_0006.nc"

However, this naming pattern shows slight deviations between data sets and over time, which is not tracible on the KNMI portal (no documentation etc.). Finding out if and where naming has changed can therefore only be done by trial and error. This makes it difficult to write a general function that can retrieve all data sets from the KNMI data portal. The function we wrote now can retrieve data for four major data sets: mean temperature, min temperature, max temperature and precipitation.

The following changes in the naming patterns of these four data sets need to be accounted for:

If such changes are known for other data sets as well, our function can easily be extended.