consider coarser resource publication with interactive links

[tomkralidis]

A WNM canonical link must be opaque such that no further interaction is required for download.

At TT-WISMD 2023-04-12, @amilan17 discussed that ECMWF has a use case where a given notification can point to a canonical link that takes further interaction from the user to be able to download data (granules). An example here could be advertising a model run with many parameters/pressure levels/timesteps.

cc'ing @sebvi for clarification (not sure the above captures things clearly enough).

[sebvi]

The question I asked is the following: is it ok to put as a link our api rather than a link to the data directly? From your first sentence it does not seem to be desirable.

We typically produce daily data i huge files containing all parameters, steps, etc. and users might not want to download these huge files so we typically point them to our api where they can download only the portion they are interested in.

[tomkralidis]

Messages can certainly add additional links for APIs, as can discovery metadata. Users who are subscribed to ECMWF data, for example, can opt to use the API in lieu of the canonical link.

[kaiwirt]

I think if you just follow the link in the WNM you should receive something meaningful. This might be a default data set in which case the link could well take additional parameters allowing to download other data sets.

Other than that we could add another type of WNM (maybe on a different topic?) specifically for API endpoints which just informs users of updates. The way they need to query the data can then be defined in the metadata for example using OpenAPI and is not part of the WNM.

[amilan17]

related to wmo-im/tt-nwpmd#13

[petersilva]

fwiw, feeds are about real-time data... processing the fraction of data available as soon as possible. NWP products for a run are likely not available at once. One would imagine publishing the products for analysis, then +6h, +12h, +18, etc... the availability would be separated by between 5-15 minutes. In addition, there may be derived/transformed products that are made available after the initial ones. If you group all the products under one announcement, can likely only announce at the end of each run, which may be upto 90 minutes after the data is available. It's not real-time anymore.

[tomkralidis]

TT-WISMD 2023-09-11:

• for core data, the data associated with a given WNM needs to be available as "one click", with no additional user interaction or specifics
• for recommended data, you could provide a notification for a new model run with the canonical link being, say, a README file, along with an additional rel=related link to further documentation, and/or pointing back to the WCMP2 document (rel=collection) with more information on how to access the data via the API (note that WCMP2 provides facilities to express URL templates for API workflow).

Specification updates:

• WNM: add example of model run notification per the above

This is a similar workflow of EUMETSAT.

@sebvi / @gaubert can you provide an example that we can include in the WNM annex of examples?

[tomkralidis]

TT-WISMD 2024-01-15:

• @gaubert to provide example for end of week
• @sebvi any update?

[gaubert]

@tomkralidis example provided in my pull request.

[tomkralidis]

Thanks @gaubert. @sebvi any chance you can also provide an example?

[6a6d74]

Jeremy (@6a6d74) will incorporate the consensus above into the Guide; section for data publishers 2.6.3

[6a6d74]

@tomkralidis - do you know which section of the WNM spec this material will go into? I'd like to reference it from the WIS2 Guide.

[tomkralidis]

@6a6d74 this would be put forth in https://wmo-im.github.io/wis2-notification-message/standard/wis2-notification-message-DRAFT.html#_links

[josusky]

I would like to emphasize that while one click to get data is nice for showcasing, it is practical only for small things like station observations. Those are a corner case in fact. Satellite, NWP and radar data are much more important and for those it is not very practical to provide a link to download global coverage with one click. Sure, in the NWP case mentioned by @petersilva it si desirable that the publisher announces data availability incrementally as the model produces individual granules, but the granules will have size in order of hundreds of megabytes. It would be very impractical to send one notification per grid point per parameter per level per forecast hour ... The subscribers of such services (and those services are the main use case) should be a bit more sophisticated than a simple "grab the URL". That is, the notifications about Satellite, NWP and radar data may have a "one click link" that points to an API/service description document (that is the README mentioned by @tomkralidis) and additional information that tells the (sophisticated) clients which forecast hour, parameter etc. has become available.