Priority

[petersilva]

in the old WMO tables, every TTAAii was assigned a priority value. Priority varied from 1 to 5, with 1 being the highest priority, and 5 the lowest (but I don't recall seeing any products at priority 5... 4 tends to be the lowest we see.)

In General the scheme is something like this:

• priority 1 messages are system messages & notifications, weather warnings, SPECIS ?
• priority 2 is observations,
• priority 3 is routine forecast products.
• priority 4 is numerical products etc...

The main criterion is the timeliness requirement for delivery of the product. I don't believe WMO describes the priority algorithm. It is understood that all priority 1 should be sent before any priority 2, 2 before 3, etc... But it isn't clear that this strict interpretation is necessary. Something which generally sends higher priority data faster than lower might be good enough (to be debated here, I suppose.)

[petersilva]

In the currently proposed message format, priority is not included. In the tables, priority is mapped from TableA.json with some values listed such as "1/2/3" and "2/4" that clearly require clarification.

[petersilva]

Approach used (past 5 years) with Canadian Stack: Identify high priority data, use a separate AMQP subscriber for it (separate queue.) so a group of subscribers are dedicated to high priority traffic, while a separate pool of subscribers download only lower priority data. The products downloaded by the high priority subscriber are selected using subtopics.

[petersilva]

in the Tokyo Cloud proposal, independent cron jobs are run to fetch data at each priority level. The lower priority cronjobs run much less frequently than the high priority ones. Neither this scheme nor the Canadian one guarantees that a priority 4 is transferred before a priority 1 in all cases, but it is very likely to be sufficient in reality.

[antje-s]

In WMO Metadata Core Profile there are 4 different priorities for global exchange on GTS: GTS priority (also known as GTS product category code) shall be expressed using the following controlled vocabulary: “GTSPriority1”, “GTSPriority2”, “GTSPriority3” and “GTSPriority4” (p.59 WMO-No 1060, https://library.wmo.int/doc_num.php?explnum_id=10141 and http://wis.wmo.int/2012/codelists/WMOCodeLists.xml, p.44 WMO-No 1061, https://library.wmo.int/doc_num.php?explnum_id=10257 )

[petersilva]

yes, we have a way of expressing it, I guess that means that if we made a JSON field for it, it would be "GTSPriority" with values 1,2,3,4. ... but most entries in the tables do not have a priority assigned, some have multiple priorities (indicating that T1 is not sufficient to determine priority) and in practice we see implementations that ignore it anyways. I would submit that priority is an implementation detail required for the GTS (when data was single streamed over a single socket socket connection) and has no reason for existence in WIS, because:

• priorities were cricual when transfers were single streamed, but all methods proposed include multiple parallel streams, and so priority does not accomplish much.
• priority is mis-implemented in existing implementations, and no-one cares/notices.
• priorities are not defined for most products, existing and not existing.
• priority assignment is ill-defined even for some projects where priority is set ( 1/2/4...)
• the priorities are, even in principle, a poor match for current needs, as they are not really universal. -- someone interested in RADAR data, that his high priority for them, but it would fall to 4 in the GTS scheme. -- someone looking for climate data, priority usually does not matter.
• clients can configure their selection criteria to achieve any prioritization scheme they desire. (We often have separate subscribers for RADAR, for warnings, for models.) we can come up with recommended criteria for members without requiring explicit priority. It is unnecessary.
• prioritisation only works well if volumes are controlled anyways. You need to have a small number of items at priority1 in order not to starve priority 2, and for priority 1 messages to pass quickly. You can achieve the same segregration and load balancing by using separate subscribers.

I think priority is, similar to the GTS' traditional "alpha" / "binary" distinction, something that is no longer a good match for technologies available, and no longer makes much sense.

[petersilva]

oh... I forgot to add: most MQP's do not have a built-in notion of priority. To implement priority, we would have to layer them on top of MQP's such as separation of subscribers and queues (using a different topic tree) or adding it to the topic hierarchy, making a knowledge of each product's priority a prerequisite for subscribing to it. The added complication to implement is more nuisance than helpful and does not accomplish anything different than what could not be done by careful topic selection without explicit priorities.

[antje-s]

I wouldn't want to set the priority in every single message, but e.g. for warnings there is a criterion that they have to be distributed within 2 minutes (https://library.wmo.int/doc_num.php?explnum_id=10257 p.11 4.4.4.2Although the cache of data and products intended for global exchange is required to be current across all GISCs to within 15 minutes, warnings must be current to within two minutes). So we should think about implementation solutions.

Maybe we could publish warnings in a separate exchange to subscribe there for warnings. Or if there are no performance problems while publishing to the exchange, subscribers could use a separate Queue...

We could also mention these possibilities as optional, only in case of performance problems

[petersilva]

some example advice in this vein: https://github.com/MetPX/sarracenia/blob/master/doc/subscriber.rst#high-priority-delivery

[petersilva]

Publishing to a separate exchange works on AMQP, but there are no exchanges in MQTT, so it ends up establishing a separate topic hierarchy for high priority messages. In either case, clients are then forced to have two (well, four?) subscribers (one for each priority) in order to get a complete data set. The same effect is achieved with four subscribers picking different topic trees (i.e. with no explicit priorities.)

This boils down to including priority in the hierarchy.It can be done, but... is it really necessary/helpful?

[antje-s]

For T1="W" there is already a separate subtree "warnings". If there are no too big delays in publishing it may be enough to point out that you should use a separate subscriber for this subtree to ensure that the warnings are received as promptly as possible. We should only keep an eye on this during the later performance evaluation...

[petersilva]

just documenting our approach. no further work expected.