Deal with GRID* files

[crdanielbusch]

I'm wary of including the GRID* files in the repo since they're not really input files nor are they part of the prior process itself but are used to create om-domain-info.nc. Can we perhaps include om-domain-info.nc itself?

Originally posted by @prayner in https://github.com/openmethane/openmethane-prior/issues/8#issuecomment-2116917137

Also see https://github.com/openmethane/openmethane-prior/pull/8#issuecomment-2117424847

[aethr]

Spoke with @prayner and @lewisjared about this today. The feeling is that it should be possible to run openmethane-prior using any domain, not just the one generated from our 10km GRID files.

I would propose two actions to resolve this issue:

1. Source om-domain-info.nc from prior.openmethane.org R2 bucket

   • [x] Generate a definitive domain-info.nc for Open Methane 10km grid using omCreateDomainInfo.py
   • [x] Upload it to the prior.openmethane.org R2 bucket (possibly under a "/domain" folder as there may be multiple in the future)
   • [x] Add the domain file to the list of files downloaded by omDownloadInputs.py in openmethane-prior
   • [x] Remove the cmaq_example folder from openmethane-prior, or replace it with a test domain file as documentation of the format

2. Move generation of the Open Methane domain file into setup-wrf

   • [x] Copy or re-create the omCreateDomainInfo.py functionality in setup-wrf, using GRID* files as output from one of the processes there
   • [x] (Optional) add a script to automatically upload the domain file to prior.openmethane.org R2 bucket using aws cli tool

[lewisjared]

Do we need to think about versioning these files at the moment? How do we deal with updates if we add additional information to the files, but that isn't a change in the actual domain.

[aethr]

Good question. I suppose the format of the domain.nc file could include a DOMAIN_FORMAT_VERSION attribute, and then the codebase could have a hardcoded value for the version it is known to work with. If the version in the nc doesn't match the codebase, we throw an error?

If you run omDownloadInputs.py and get a new domain.nc with a newer version, we throw an error saying "please update openmethane-prior to the latest version".

If you update openmethane-prior and the version is newer than the nc file you have, we throw an error saying "please run omDownloadInputs to fetch a more recent domain file"?

[prayner]

The "bad" case is where something new in the grid file changes a behaviour of existing gcode. but everything still runs. Adding new features in code that require new info in the grid files is fine I think and the more likely case. So what we really want to know is the version of the code that created the file. Can git help here?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

-- Peter Rayner (he/him), Honorary Professorial Fellow, University of Melbourne mobile +61 402 752 379 zoom id 4431343191, join at https://unimelb.zoom.us/j/4431343191?pwd=a1E5Z3JEOTRVQUJsaVdRbVUvR1QyZz09 mail-to: @.*** google scholar: https://scholar.google.com.au/citations?user=H3up71wAAAAJ&hl=en I acknowledge the Traditional Custodians of the land on which I work, the Wurundjeri people of the Kulin nation, and pay my respect to their Elders, past and present I am sending this email when convenient for me, please only respond when convenient for you

[aethr]

In software this would be handled with semver versions, with versions like "1.2.3". ie "MAJOR.MINOR.PATCH".

• increasing MAJOR version means breaking changes (your "bad" case)
• increasing MINOR version means new features/changes but no change to existing
• increasing PATCH typically means a bug fix or no-op change

Not sure we need to go that far. For compat we probably only need to manage the "MAJOR" case, and even that might be overkill before we release to the public.

[lewisjared]

The domain generation code has been moved to the openmethane repo now. The current config uses the domain name and version to download the appropriate input domain