search

MetOffice / localTables-GRIB-BUFR

management of local tables for WMO GRIB and BUFR formats
3 stars 7 forks source link

local table version #40

Open mo-marqh opened 3 years ago

mo-marqh commented 3 years ago

GRIB specification requires a local table version number

missing (255) causes issues with some software

How does the codes registry enable a fixed published version to be presented and obtained, whilst maintaining a current set of entries spanning versions?

mo-marqh commented 2 years ago

this is release, not so much a version

mo-marqh commented 2 years ago

Suggested trigger / control

a new control file is provided to manage a list of releases, including date of release and any ommitted entities

turtle content is created from this control file - once only

highly recommended that a repository tag is created upon successful merge, matching the release name, to enable traceback if required (this can be done from git history, but more convoluted)

Suggested registry structure

Releases exist, are defined, within the name space of the register, so

https://reference.metoffice.gov.uk/grib/grib2/mo--74/mo--74-1 is the release 1 of https://reference.metoffice.gov.uk/grib/grib2/mo--74

(this brings some content maintenance challenges which need to be mitigated)

Registers are created to match the content structure of the main published content https://reference.metoffice.gov.uk/grib/grib2/mo--74/mo--74-1/4.5

RegisterItems for each entity in the main content are created, to match the register structure of the published content

but, RegisterItems do not contain a new Entity, instead they reference an existing entity

thus the RegisterItem

https://reference.metoffice.gov.uk/grib/grib2/mo--74/mo--74-1/4.2/_0-7-193 references the Entity https://reference.metoffice.gov.uk/grib/grib2/mo--74/4.2/0-7-193 there is no published entity https://reference.metoffice.gov.uk/grib/grib2/mo--74/mo--74-1/4.2/0-7-193 the registry treats this as a known null entity, not the published quantity

mo-marqh commented 2 years ago

https://github.com/UKGovLD/registry-core/wiki https://github.com/UKGovLD/registry-core/wiki/Principles-and-concepts#register-registeritem-entity https://github.com/UKGovLD/registry-core/wiki/Api-example%3A-registration

mo-marqh commented 2 years ago

56

feggleton commented 2 years ago

Feedback email trail to discuss before going forward:

"Hi all,

Apologies for not sending more of an explanation with my request! Perhaps a demo/chat about this when Mark returns would be better to describe the process. Essentially what Tom has described is correct bar a couple of things. We would release version 1 as the local tables currently stand so I don’t think there would need to be a check they would be an exact copy. That release however would be static and once published never change. Then if new things are added they will be added to the dynamic version you see on the front page under mo—74 (not mo—74-1) and eventually a new version will be published with any new additions or deprecations. So from my understanding the codes which have been published in the WMO table may show as active in the old version release (if they were active at the time) but show as deprecated in the dynamic tables and any newer release version after this change has taken place.

As I write this I realise this will probably need rethinking as you’ve mentioned there would be a lack of release number for the dynamic version so thanks for raising that concern. We need to think about what happens between versioned releases being published. I think it might be best to discuss this with Mark when he’s back as he knows the process better than me and we can work with you guys to get something that works for your systems too (hence my request for feedback before going any further). I may have also completely mis-interpreted Marks idea so best to confirm when he’s back! Thanks so much for clarifying your end of this process that’s really helpful 😊

Food for thought – with the CF Conventions Standard Names Table we only publish a new version every few months so we suggest to people to start using the terms (once agreed) before the new version is published with the next incremented version number so when it is eventually published the files are correct..

Thanks,

Fran

From: Powell, Thomas [thomas.powell@metoffice.gov.uk](mailto:thomas.powell@metoffice.gov.uk) Sent: 30 June 2022 18:19 To: Salter, James [james.salter@metoffice.gov.uk](mailto:james.salter@metoffice.gov.uk); Eggleton, Francesca [francesca.eggleton@metoffice.gov.uk](mailto:francesca.eggleton@metoffice.gov.uk) Cc: Hedley, Mark [mark.hedley@metoffice.gov.uk](mailto:mark.hedley@metoffice.gov.uk) Subject: RE: GRIB2 versioning review

I think I understand what Mark has done here….

So, what needs to be done… MDDA side - confirm the version Mark has created contains all the codes we currently use in our data products and feedback to DM team if not. Anything not there should be added. DM side -

  1. As James says, any local code which is now published in WMO tables should be tagged as ‘deprecated’ in the mo—74-1 version with a pointer to the stable WMO code.
  2. Once the above is complete these codes can be either marked as deprecated in the unversioned tables or removed altogether.
  3. Once confirmation is received from MDDA on the other codes, Fran/Mark should re-label the ‘released’ mo--74-1 from ‘experimental’ to ‘stable’ (and everything within other than the above)
  4. Once the above is done, the current ‘stable’ tables should be tagged as ‘experimental’ - This allows us to have a v1 plus a development version for anything new.

Once released we should pass some kind of comms to users about what we’ve done so they can update any docs they have accordingly.

This seems pretty good to me, but I don’t know what we’d do if we were adding a new local grib code after the release of v1… AFAIK you can only enter integer values in the local grib codes version field within data products, so if we wanted to reference new codes not in v1 what do we put? Maybe 2? Q for DM team to advise on 

Tom

From: Salter, James [james.salter@metoffice.gov.uk](mailto:james.salter@metoffice.gov.uk) Sent: 30 June 2022 16:14 To: Eggleton, Francesca [francesca.eggleton@metoffice.gov.uk](mailto:francesca.eggleton@metoffice.gov.uk); Powell, Thomas [thomas.powell@metoffice.gov.uk](mailto:thomas.powell@metoffice.gov.uk) Cc: Hedley, Mark [mark.hedley@metoffice.gov.uk](mailto:mark.hedley@metoffice.gov.uk) Subject: RE: GRIB2 versioning review

Hi Francesca,

To be honest I’m not sure what I’m looking at here and the feedback you want me to give.

Is the ‘experimental’ the new version? Which will become stable, or something else? If this is the case, the first thing I checked was the now deprecated codes of 0-3-192 & 0-3-193, as now adopted by WMO – I would expect these to go from a later version of MO local codes. As I said, not entirely sure what I’m looking at, so this may not be relevant feedback.

Cheers James

From: Eggleton, Francesca [francesca.eggleton@metoffice.gov.uk](mailto:francesca.eggleton@metoffice.gov.uk) Sent: 30 June 2022 12:11 To: Salter, James [james.salter@metoffice.gov.uk](mailto:james.salter@metoffice.gov.uk); Powell, Thomas [thomas.powell@metoffice.gov.uk](mailto:thomas.powell@metoffice.gov.uk) Cc: Hedley, Mark [mark.hedley@metoffice.gov.uk](mailto:mark.hedley@metoffice.gov.uk) Subject: GRIB2 versioning review

Hi all,

Mark has been working on an idea for releasing versions of the GRIB2 MO local code tables and we would like for you to review this and let us know any feedback including if it would work for you in MDDA. Mark is away the whole of July so no changes/implementation will be made until he returns. We have pushed the changed up to the codes registry site under ‘experimental’ status so you can review:

http://reference.metoffice.gov.uk/grib/grib2/mo--74 There is 1 known issue we are aware of relating to unexpected entity creation, reported UKGovLD/registry-core#159 Thanks,"