Organize level 8+ sub-topics

[amilan17]

Levels 1 - 7 are organized as flat CSVs, but at some point the sub-topics will need to branch off. For discussion, below are some screenshots of the current organization and some alternative options.

[amilan17]

Levels 1-8, represented as flat CSVs.

[amilan17]

Levels 8+ as described by the Unified Data Policy with domain specific input (so far) highlighted in yellow. 

[amilan17]

Option A, following current approach above, suggests that we maintain consistency at the same levels wherever possible.

[amilan17]

Option B, branching occurs at level 9 as "datatype or category" for predictions, observations, advisories-warnings. 

[tomkralidis]

TT-WISMD 2023-05-10:

• go with option A (consistent sub-category under each Earth System Discipline [level 9])
• add common subtype under earth system discipline:
  • surface-based-observations
  • space-based-observations
  • advisories-and-warnings
  • predictions (whether to subqualify or not?)

...but allow domains to request additional common subtype as needed.

• need to verify validation of topics complexity from GB (@golfvert)

[amilan17]

@alexandreleroux recommends having the top level be "observations" and the type under that (space, ground, surface)

[golfvert]

Up to level 8 and some 9 (when available in Res. 1), we want to stick to Res. 1 denomination. I don't think this proposal is consistant with this.

[amilan17]

see draft at: https://github.com/amilan17/wis2-topic-hierarchy/tree/main/topic-hierarchy

[amilan17]

Up to level 8 and some 9 (when available in Res. 1), we want to stick to Res. 1 denomination. I don't think this proposal is consistant with this.

In my opinion, UDP is too inconsistent to follow exactly. For example, "space-based" and "surface-based" are only sub-categories under weather and space-weather disciplines, but satellite data are also listed for hydrology and cryosphere disciplines, but not under any sub-category. Furthermore, I find it hard to believe that core satellite data may not apply at all under atmosphere, ocean or climate.... 

For reference, see Annex 1 of WMO Unified Data Policy: https://library.wmo.int/index.php?lvl=notice_display&id=22100

[tomkralidis]

TT-WISMD 2023-06-22:

• @amilan17 / @antje (and possible @jsieland / @gaubert) will prepare something for the next meeting (2023-07-07)

[antje-s]

...wondering if it is really not consistant to add a level "observations" with WMO Unified Data Policy (https://library.wmo.int/doc_num.php?explnum_id=11256 , Annex 1)

1. Weather-related data 1.1 Core observational data 1.1.1 Surface-based 1.1.2 Space-based

e.g. origin/a/wis2/deu/dwd/data/core/weather/surface-based-observations/ update to origin/a/wis2/deu/dwd/data/core/weather/observations/surface-based/

[yhe-wmo]

update to origin/a/wis2/deu/dwd/data/core/weather/observations/surface-based/

This is indeed a recommendation put forth by TT-NWPMD as well, in order to maintain a simple and concise structure on Level 9.

[golfvert]

update to origin/a/wis2/deu/dwd/data/core/weather/observations/surface-based/

We could.

Looking at hydrology in the mind map above:

If we create an observation level, what do we do with the other two categories ?

The real (only ?) goal of the topic hierarchy is to have a filtering mechanism to receive only useful messages. With MQTT we have two wildcards ? (anything at this level) and # (anything below this level). With this in mind, from a user point of view and management of subscription point of view, what is the "best" option.

origin/a/wis2/deu/dwd/data/core/weather/surface-based-observations/
or
origin/a/wis2/deu/dwd/data/core/weather/observations/surface-based/

Looking again at the 4 categories for hydrology, what is useful to filter for a user ? The 4 at the same level ? 2 at one level and two below observation ? For me, these are the key questions...

[antje-s]

Suitable filter options for the different use cases is relevant, I absolutely agree with that. But from my point of view, there is nothing to be said against it for this reason as well....

new TH: .../weather/observations/surface-based/ .../weather/observations/space-based/ .../weather/analysis-prediction/ .../weather/advisories-warnings/

old TH: .../weather/surface-based-observations/ .../weather/space-based-observations/ .../weather/analysis-prediction/ .../weather/advisories-warnings/

But maybe I'm overlooking some use cases, the ones I would think of are: 1) all observations subscribe to new - .../weather/observations/# old - .../weather/surface-based-observations/# & .../weather/space-based-observations/# (to avoid simultaneous subscription to .../weather/analysis-prediction/ and .../weather/advisories-warnings/) +1 for new

2) only surface-based-observations new - .../weather/observations/surface-based/# old - .../weather/surface-based-observations/# even though the new one has one more level, the total string length is hardly longer, so i would rate both versions as equal

3) only space-based-observations see 2

4) for analysis-prediction & advisories-warnings no difference

5) all messages under weather new - .../weather/# old - .../weather/# no difference

[amilan17]

Tag up 2023-07-05 (w/ Antje, Julia and Anna)

• (Antje) notifications about services should be classified as 'report' and 'service' is not needed, because it's only information for the users
• prefer to have observations (9) with the space-based, surface-based (10)) as options underneath, because users can filter out level 10 if they don't care if it's space-based or surface -based
• agreed that consistency at level 9 for a minimum set and maybe level 10 under observations

Level 9 (sub-discipline type)

• observations
• analysis-predictions
• advisories-warnings

Observations > Level 10 (need a name)

• surface-based
• space-based
• ground-based (for hydrology)

Level 10 under analysis-predictions

• prediction types (analysis, reanalysis, forecasts...etc)

Climate (8)

• observations (9)
  • (10) space/surface based
  • (11) monitoring ?
• analysis-predictions (9)
  • prediction-type > reanalysis

[tomkralidis]

TT-WISMD 2023-07-07:

• @amilan17 to create PR based on the approach (will not include climate)

[amilan17]

PR #53 includes decisions discussed in July tag up: https://github.com/wmo-im/wis2-topic-hierarchy/issues/38#issuecomment-1621604073

for discussion:

1. Is it ok to have flat levels (1-7) and then nested levels (8+)? It feels like we are changing the rules mid-game.... 
2. Should every level have a CSV listing the child folders? This is necessary for committing "empty" folders in GH but is it useful for other things too? (see https://github.com/wmo-im/wis2-topic-hierarchy/tree/38-organize-level-8%2B-sub-topics/topic-hierarchy/weather vs https://github.com/wmo-im/wis2-topic-hierarchy/tree/38-organize-level-8%2B-sub-topics/topic-hierarchy/atmospheric-composition)
3. Child column is removed from nested levels, do we still need it?
4. Do we need definitions?

[tomkralidis]

TT-WISMD 2023-09-12:

• PR #53 to be reviewed and approved/merged thereafter (@amilan17 to update)

[amilan17]

This approach simplifies the management of levels, allows for mixing/matching instead of adding duplicate values under new levels, and moves the conversation away from developing an ontology

[golfvert]

Are we saying that any combination of elements from each level is a valid topic ? So .../data/core/climate/observations/forecast/green-house-gases is acceptable (I chose on purpose the weirdest combination).

[amilan17]

comments from meeting with @efucile:

• We need the tree approach because if people can mix/match in a nonsensical way, they will

[SimonElliottEUM]

We need a tree approach to cover the diversity of data destined for WIS 2.0. As @golfvert and @efucile mention, we leave the door open for nonsense on one hand (.../data/core/space-weather/observations/surface-based/...), and preclude precision on the other

[efucile]

Decision

Topic hierarchy is proposed by domain groups to TT-WISMD and considered for consistency across domains. Approval of new topic hierarchy will be by fast-track. Valid topic hierarchies will be published on an authoritative publication. GDC will check that only topic hierarchy from the authoritative list are published. Non valid topic hierarchy will be rejected (monitoring action to be decided). GB will query the GDC api to get a list of published topics (this will be all valid according to the authoritative list of allowed topics). GB will subscribe to the list of valid topics published by the GDC to process the notifications from the WIS2 nodes. Experimental data can be published under experimental topic which has been added under all the domains. Under experimental topic level all sub-topics are allowed and GDC will not apply any check. Experimental topics are meant to be temporary and they shall be converted to an officially approved topic in the authoritative list of topics.

[tomkralidis]

Associated PR now in #70