Define endpoint and capabilites for metadata discovery and search

Teddy-1000 commented 7 months ago

Summary originally written by @GijsbertKruithof . We had a meeting today 15-02-2024 with @Phaf, Lukas (KNMI), @amundi@met.no, @Jo Asplin and @jukka.pelli@fmi.fi about how to provide metadata via the API.

Especially if we do this with an OGC Records API or with the OGC EDR API.

Outcome

We can use the EDR Items endpoint for time series metadata
The discovery metadata of the datasets will be made available via a JSON file. The message from the MQTT broker will point to an url where the JSON file can be fetched.
For station metadata the user will be pointed to a OSCAR entry, which will be based on the WIGOS identifier – if present in the metadata.

Main reasons for using EDR:

The EDR Items endpoint has exactly the same features structure in both API’s. EDR: /collections/{collectionId}/items, OGC Records: /collections/{collectionid}/items But using the OGC EDR is easier for the user to understand, and easier for us to make and maintain

To be done:

[x] - The time series metadata needs to be added to the EDR API to be searchable.
[x] - The WIGOS identifier needs to be translated into an URL in the API output
[x] - The discovery metadata on the Dataset needs to be made available in a JSON file.

GijsbertKruithof commented 7 months ago

Hereby the first version of the Discovery Metadata file knmi-10min-observations.json

Teddy-1000 commented 7 months ago

I will start implementing the search for time series in the EDR API. Any thoughts on where we should put the dataset Discovery Metadata file. I don't think it can be under /items?

Teddy-1000 commented 7 months ago

After some looking in to features vs Records items search, the main difference seems to be the support CQL querying. In a features endpoint, we need to add every parameter we want to be searchable. In records we would rather implement a CQL2 expression for filtering. Both are of course possible to implement, any thoughts on which we should go with? @lukas-phaf @jo-asplin-met-no @johtoblan

vegark73 commented 6 months ago

Could this be a first atempt for the text required?

"title": "Additional and subhourly observations from European area, including synoptic observations distributed over GTS",

"description": "collection of observations from weather stations situated in European countries. Data from synoptic stations and additional stations from 3. party station holders is distributed with best possible time resolution. All parameters resolved in the CF-standards could be distributed. The number of parameters differs pr station. Observations from the last 24hours is available for download from EDR-api. Timestamp for each observations is based on observation time, or the end of a aggregation period. Depending om the method used to produce the parameter. All timestamps is given as UTC. Timeliness of the data depends on when it is distributed from the NMS. Timeliness within the system is 1 minute after receprion time at the E-soh ingestor. Observations in E-soh is initialy received at one of the NWS'es. The NMS is responsible for checking the quality of the observations. The NMS is also responsible for collection and distribution of metadata compliant to E-soh and WIS-2.0. E-soh also include all global observations distributed over the Global Transition System (GTS) that is considered as open and free for use. E-soh aims to serve all data according to FAIR principles.

For questions or technical documentation, you can go to https://?????????

GijsbertKruithof commented 6 months ago

I updated the example metadatafile for the KNMI dataset after comments from Jeremy Tandy and Tom Kralidis. nl-knmi-test-Actuele10mindataKNMIstations-2-NEW.json

EURODEO / e-soh