wmo-im / wmdr

WIGOS Metadata Standard: UML model and schemas
7 stars 7 forks source link

Suggestion to review the WMDS element 7-14 "Schedule of International Exchange" #39

Open nuneslf opened 4 years ago

nuneslf commented 4 years ago

Rationale: The WMDS should allow for three levels of schedule: i) observation (6-08 Schedule of observation), ii) reporting to a central location, e.g. NMHS headquarters (7-03 Temporal reporting period) iii) international reporting (7-14 Schedule of international exchange) When ii and iii are the same, it is okay that 7-14 is a binary element (Y/N); the issue is when they are different, then 7-14 needs to be allow providing the actual schedule.

Element 7-14 was introduced to distinguish international reporting from non-international reporting, i.e. from the first level of reporting, for which 7-03 is used; So, in the WMDS element 7-14 should be "detached" from the non-international level of reporting to allow describing a different schedule if needed. Example: an AWS is configured to collect data every minute all-year-round (schedule of observation); it sends data to a central system every 30 minutes; reports are disseminated to GTS/WIS every hour.

Below are the current WMDS definition and note, with some suggestions: Current definition = Allows distinction if/when observations are made/not made available internationally; Suggested new definition = Schedule of observations reported internationally Current NOTE: A binary element (yes/no), that should be associated with the observed variable for each declared observing schedule (7-03 Temporal reporting period) Suggestion to remove the current NOTE.

joergklausen commented 3 years ago

The DataTypes "Schedule" (which is really a coverage) and "Reporting" are currently mandatory attributes of the FeatureType "DataGeneration" (cf. https://schemas.wmo.int/wmdr/1.0RC9/html/). So, logically, for each DataGeneration validPeriod, a Schedule and Reporting need to be defined. This allows for changes over time. The example given by @nuneslf can be partially translated using elements as follows:

If a more explicit differentiation between national and international reporting is really needed, it appears that changing the cardinality of the "Reporting" attribute of DataGeneration from currently 1 to 1..2 is the most appropriate way to go. The existing flag internationalExchange would indicate if exchange is international or not. This would also create the flexibility to specify different dataFormat, dataPolicy, officialStatus etc. for national and international reporting. A specific recipient would still not be specified.

@nuneslf and other interested stakeholders: Please comment if this meets the requirements.

nuneslf commented 3 years ago

If I understand correctly the "wmdr schema" language used in the comment by @joergklausen, I'd say that the main point is not about the reporting target (WIS or something else), the issue comes if/when the frequency and/or aggregation is different between the 2nd to the 3rd bullet above, i.e. from the central (national) collection (every 30' in the example) to the international exchange (every 60' in the example). The current configuration doesn't allow distinguishing these two.

KarlBureau commented 3 years ago

Hi all

I'm just parachuting into this topic without knowing all the context. So ignore anything irrelevant.

Using our Australian AWS as an example:

  1. Sensors measure every second.
  2. A 1 min message is constructed using valid 1s samples (Avg, Max, Min, SD) (original 1 second data in local memory is overwritten in the field equipment after a couple days)
  3. 1 min message is sent to central system every minute (AWS reporting schedule)
  4. 1 min data is used to develop the following products:
    • 10 min message
    • METAR/SPECI, nominally 30 min (though a SPECI can be generated at any minute)
    • SYNOP generated hourly;
  5. National reports
    • Internal use and some specific customers) - 1 minute data
    • 10 min for website and fire weather products
    • Half-hourly and hourly products
  6. WMO reports
    • Hourly Synop (some reports on half-hour as two states have a 30min time zone offset)
    • Half-hourly METAR

Does that give you a tangible example to test the scheduling specification?

Regards Karl


Karl Monnik | Manager Data Requirements & Quality | Data Program | Data & Digital Group | Bureau of Meteorology T: +61 (0)3 9669 4205 | M: +61 (0)4 1329 4772 (Pvt) | @.**@.> | www.bom.gov.auhttp://www.bom.gov.au/


Important: This message may contain confidential or legally privileged information. If you think it was sent to you by mistake, please delete all copies and advise the sender.

From: Jörg Klausen @.> Sent: Friday, April 23, 2021 7:46 PM To: wmo-im/wmdr @.> Cc: Karl Monnik @.>; Assign @.> Subject: Re: [wmo-im/wmdr] Suggestion to review the WMDS element 7-14 "Schedule of International Exchange" (#39)

The DataTypes "Schedule" (which is really a coverage) and "Reporting" are currently mandatory attributes of the FeatureType "DataGeneration" (cf. https://schemas.wmo.int/wmdr/1.0RC9/html/https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fschemas.wmo.int%2Fwmdr%2F1.0RC9%2Fhtml%2F&data=04%7C01%7Ckarl.monnik%40bom.gov.au%7C7f8a79822e15445d82a708d9063ca77c%7Cd1ad7db597dd4f2b816e50d663b7bb94%7C0%7C0%7C637547679840958173%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DAtTjX7rEzJ0KB2o%2BKdaJi14BSHmDuIZjrpVK7%2BcCa0%3D&reserved=0). So, logically, for each DataGeneration validPeriod, a Schedule and Reporting need to be defined. This allows for changes over time. The example given by @nuneslfhttps://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnuneslf&data=04%7C01%7Ckarl.monnik%40bom.gov.au%7C7f8a79822e15445d82a708d9063ca77c%7Cd1ad7db597dd4f2b816e50d663b7bb94%7C0%7C0%7C637547679840958173%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GlQ1gdP1dlTNUWHQfl66T8q%2FZSCYfVsodz0G1E8n2OQ%3D&reserved=0 can be partially translated using elements as follows:

If a more explicit differentiation between national and international reporting is really needed, it appears that changing the cardinality of the "Reporting" attribute of DataGeneration from currently 1 to 1..2 is the most appropriate way to go. The existing flag internationalExchange would indicate if exchange is international or not. This would also create the flexibility to specify different dataFormat, dataPolicy, officialStatus etc. for national and international reporting. A specific recipient would still not be specified.

@nuneslfhttps://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnuneslf&data=04%7C01%7Ckarl.monnik%40bom.gov.au%7C7f8a79822e15445d82a708d9063ca77c%7Cd1ad7db597dd4f2b816e50d663b7bb94%7C0%7C0%7C637547679840968166%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UFQdK8UsJHEFzRjaaDwq5I%2BNDd%2BDm9FnNdd3Ea4pEqs%3D&reserved=0 and other interested stakeholders: Please comment if this meets the requirements.

- You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwmo-im%2Fwmdr%2Fissues%2F39%23issuecomment-825539620&data=04%7C01%7Ckarl.monnik%40bom.gov.au%7C7f8a79822e15445d82a708d9063ca77c%7Cd1ad7db597dd4f2b816e50d663b7bb94%7C0%7C0%7C637547679840968166%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=IvVLLjuNk3edL5SH44rGzlVrOYofDULT43YJnAIy1SA%3D&reserved=0, or unsubscribehttps://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAK22QL4VJPWNGWTYCSHOE53TKE6VXANCNFSM4WRLC63A&data=04%7C01%7Ckarl.monnik%40bom.gov.au%7C7f8a79822e15445d82a708d9063ca77c%7Cd1ad7db597dd4f2b816e50d663b7bb94%7C0%7C0%7C637547679840978157%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OTiAG1R3xZbIfDye4h6PE1GN7M%2FYP7nOvL8FumN%2BwPs%3D&reserved=0.

amilan17 commented 1 year ago

https://github.com/wmo-im/tt-wigosmd/wiki/2023.02.03-TT-WIGOSMD notes: this is one of the most requested changes to the standard.

david-i-berry commented 1 year ago

I may be missing something obvious but is this not covered by having multiple DataGeneration entries for a given instrument.

In principle there should be one entry for the data generation and transmission from sensor to a central location (e.g. 5 second data), one for the national exchange of e.g. 10 minute data and then a final one for the international exchange (e.g. hourly data). There will be processing applied at each of these steps, different software, different aggregation periods, different formats etc. The ability to do this is covered by the existing data model (the capabilities of OSCAR Surface may be different), see figure below:

image

david-i-berry commented 1 year ago

Here's an example from https://oscar.wmo.int/surface/#/search/station/stationReportDetails/0-20000-0-06791 where the above is used in practice, we have hourly reporting for internationally exchanged data, 10 minute reporting for non internationally exchanged data.

image

joergklausen commented 1 year ago

Cf. https://github.com/wmo-im/wmdr2/wiki/WMDRRoadmap where this is taken up as well.