search

wmo-im / wcmp2

WMO Core Metadata Profile 2
https://wmo-im.github.io/wcmp2
6 stars 3 forks source link

WIGOS metadata integration: referencing a set of stations in a WCMP2 record based on a network of stations #19

Open tomkralidis opened 2 years ago

tomkralidis commented 2 years ago

@wmo-im/tt-wismd @wmo-im/tt-wigosmd @joergklausen @steingod @thineshsornalingam

Consider a WCMP2 record describing, say, surface weather observations from a network of stations/facilities. In WCMP2, we are able to enumerate 1..n links, i.e:

(NOTE: relation types are examples/for brainstorming)

{
    "rel": "http://www.wmo.int/def/rel/wmdr/1.0/ObservingFacility",
    "type": "application/xml",
    "title": "Station report for Thessaloniki (Greece)",
    "href": "https://oscar.wmo.int/surface/rest/api/wmd/download/0-20008-0-THE"
}

In the above, a user can discover a dataset collection and determine from the link relation (rel) the association station(s) providing the dataset. Now imagine a observing network with 800 stations.

WCMP2 requires an approach whereby a data provider can provide a facility set as a single link in order to scale.

{
    "rel": "http://www.wmo.int/def/rel/wmdr/1.0/FacilitySet",
    "type": "application/xml",
    "title": "Station report for facility set 123",
    "href": "TBD"
}

Questions:

  1. what do folks think about this approach? This would require data providers to define such sets in OSCAR/Surface
  2. can the OSCAR/Surface API provide a link a facility set?

Aside: while the above examples use the OSCAR/Surface API, in theory the value of the href is opaque here, so long as a WMDR facility set is provided in the response.

Looking forward to this important discussion.

efucile commented 2 years ago

There is already a requirement to identify the stations as being part of GBON for a country. Not sure if this will be accessible through the API in a convenient way. @kurt-hectic can you give some info on this?

joergklausen commented 2 years ago

The OSCAR/Surface API is a non-standard API, so in principle, parameterized endpoints can be defined ad libitum. The present API supports direct access

See User Manual for reference.

I believe, the endpoint to expose station sets is going to be available later in 2022.

The question could be if an OGC standard API like STA should be supported in future.

tomkralidis commented 2 years ago

Thanks @joergklausen. It would be valuable to have the facility set use case realized in the API. As an aside, we would be interested in discussions around OSCAR API future plans, and would suggest that the OGC API suite of standards is considered.

tomkralidis commented 1 year ago

To summarize an option:

Whether to choose option i or ii above depends on the number of stations that a given dataset is based on. For example, for ~800 surface stations in Canada, we would use option 2.

In both cases:

tomkralidis commented 1 year ago

TT-WISMD 2023-05-05: ACTION @wmo-im/tt-wismd to review and provide comments.

josusky commented 1 year ago

@tomkralidis your summary matches what we have discussed earlier and there is little I can add to it. Clearly, option 2 will be a great help and the only open question is if the existing OSCAR API station filtering capability is able to cover all cases. And if not, whether the right way is to ask for its extension or whether it is then the task of the publisher to provide an API (possibly a facade over OSCAR) that will return the right list of stations.

steingod commented 1 year ago

@tomkralidis I have nothing to add to what you commented on. Although I doubt that station metadata are managed in OSCAR/Surface I would assume many provide a copy to OSCAR where it is published and made available. But that is not the major issue here. As long as both options for linking WIS and WIGOS are possible it is positive although I find the option of direct relation between station dataset (WIS) and station metadata the most useful for services targeting decision support through integration in decision support systems at various levels. It will be interesting to see if WMO information services will be embraced in this context.

tomkralidis commented 1 year ago

Thanks for the feedback. At this point we'll add an example WCMP2 record to this effect that will become a curated example as part of an Annex in WCMP2.

tomkralidis commented 1 year ago

For station sets, given implementation in OSCAR/Surface is pending next generation discussions, perhaps an additional option can be to point to a list of stations in any format:

{
    "rel": "related",
    "type": "text/csv",
    "title": "Dataset station list",
    "href": "https://dd.weather.gc.ca/observations/doc/swob-xml_station_list.csv"
}

Or:

{
    "rel": "related",
    "type": "application/json",
    "title": "Dataset station list",
    "href": "https://example.org/oapi/collections/stations/items?f=json"
}

Note: we could (in WCMP2) perscribe a dedicated link relation (e.g. http://wis.wmo.int/metadata/stations) for tighter integration with WIS2 tooling, at the cost of broader interoperability.

joergklausen commented 1 year ago

Even the current implementation of station clusters / station ets / FacilitySet in OSCAR/Surface exposes a URL, e.g., https://oscar.wmo.int/surface/#/search/stationClusterReport/5. Since there is no WMO/WIGOS identifier for this, the identifier used (here: 5) will probably not survive the current OSCAR/Surface implementation, though. The requirement has been noted for OSCAR NextGen.

amilan17 commented 1 year ago

https://github.com/wmo-im/tt-wismd/wiki/Meeting-2023-08-14 notes:

Tom: should we include a section in the Guide for this or should this be a part of best practices? 

tomkralidis commented 1 year ago

TT-WISMD 2023-08-14

tomkralidis commented 1 year ago

Initial link relations put forth in https://github.com/wmo-im/wcmp2-codelists/pull/4

We will need to discuss whether this is the best approach (would this be better managed in WMO Common Code Tables?), as well as the publication to codes.wmo.int (which would then be the canonical reference via infrastructure).

cc @david-i-berry

tomkralidis commented 11 months ago

Notes following discussion with ET-WTR (cc Cristina Prates @thineshsornalingam):

We want to reduce duplication between WIS2 and OSCAR. Notes:

The result would be a requirement or recommendation on GDCs to augment WCMP2 on ingest per the above.

Considerations:

tomkralidis commented 11 months ago

Updated examples based on new link relations:

Single facility:

{
    "rel": " http://def.wmo.int/def/rel/wmdr/-/EnvironmentalMonitoringFacility",
    "type": "application/xml",
    "title": "Station report for Thessaloniki (Greece)",
    "href": "https://oscar.wmo.int/surface/rest/api/wmd/download/0-20008-0-THE"
}

Facility Set

{
    "rel": "http://def.wmo.int/def/rel/wmdr/-/FacilitySet",
    "type": "application/xml",
    "title": "Station report for facility set 123",
    "href": " https://oscar.wmo.int/surface/rest/api/facilityset/123"
}
tomkralidis commented 11 months ago

TT-WISMD 2023-09-13:

tomkralidis commented 10 months ago

TT-WISMD 2023-10-06:

amilan17 commented 10 months ago

WMDR codes currently only has codes for Facility Type, should there be a request to expand this code list or create a new codes list in WMDR to support this type of linkage? 

https://codes.wmo.int/wmdr/FacilityType

amilan17 commented 10 months ago

We will need to discuss whether this is the best approach (would this be better managed in WMO Common Code Tables?), as well as the publication to codes.wmo.int (which would then be the canonical reference via infrastructure).

Alternatively, I like the idea of using common code tables, but it's currently linked to WMO-No. 306, vI.2 https://codes.wmo.int/common

tomkralidis commented 10 months ago

Current list in https://github.com/wmo-im/wcmp2-codelists/blob/main/codelists/link-type.csv

Options:

  1. define entities in each specification, and publish to codes.wmo.int
  2. define in a "common/shared" vocabulary a. WMO 306, or something else) b. another vocabulary (WIS2 concepts?)
  3. other options?

Note that WMO 306 (https://github.com/wmo-im/cct) is for data specific code tables, whereas we are trying to describe higher level things.

amilan17 commented 10 months ago

@joergklausen may have ideas on how to manage shared vocabularies across WIS and WIGOS metadata....

joergklausen commented 10 months ago

WIGOS was born to cover common topics of interest, specifically for observations. So, I am fully in support of sharing common vocabularies. In fact, we should clarify what we mean by common ... between whom? The respective metadata models shold refer to the correct name space, which for WIGOS doesn't always have to be wmdr. Of course, we need clean governance structures.

amilan17 commented 10 months ago

https://github.com/wmo-im/tt-wismd/wiki/Meeting-2023-10-23 notes:

tomkralidis commented 10 months ago

I started drafting a section in WCMP2 (clause "stations/observations"), and I'm conflicted. If we add a specific section for "stations/observations", does this introduce precedent to add more sections for NWP, Hydrology, Cryosphere, etc.?

Options:

  1. in WCMP2 proper, introduce a section foreach discipline, starting with "stations/observations" (over time)
  2. discipline specific information goes into a "best practice/guidance" document (which we will start after the INFCOM-3 deadline). For INFCOM-3, build out example documents (for "stations/observations" and other disciplines). If specific elements/properties are added to WCMP2 proper, then they are obviously added accordingly

Thoughts? cc @antje-s @jsieland @josusky @Amienshxq @solson-nws @gaubert @amilan17 @masato-f29 @McDonald-Ian

tomkralidis commented 9 months ago

TT-WISMD 2023-11-16: add similar example from OceanOPS, for example.