Retrieve provenance information from CMR

doug-newman-nasa commented 1 year ago

Populate the meta.yaml file of a recipe with provenance information from the collection record in CMR.
Retrieve the short name and version id from meta.yaml (if present) and use that to query CMR. This way, the provenance of the input for a recipe is discernible from the configuration in a known location, rather than the code.

Given the above, I'm leaning to an implementation where we have a class constructor that figures out the input collection and/or retrieves the provenance information for a collection and populates the yaml file. Then users invoke the get files method. For example:

instance = PangeoForgeCMR('MOD02QKM', '6.1')
file_pattern = instance.files_from_cmr(nitems_per_file=1,
        concat_dim='time')

Alternatively, if the short name and version id are to be extracted from meta.yaml

instance = PangeoForgeCMR()
file_pattern = instance.files_from_cmr(nitems_per_file=1,
        concat_dim='time')

Where the yaml file contains,

provenance:
  ...
  source_data:
    cmr:
      short_name: GPM_3IMERGDL
      version: '06'

Please give your thoughts on this API suggestion before I implement!

yuvipanda commented 1 year ago

Thanks for opening this up! /cc @cisaacstern to see what he thinks. I'll try to come back and provide a more whole response in a day or so!

cisaacstern commented 1 year ago

Thanks for thinking through this @doug-newman-nasa, and for the tag @yuvipanda. Some thoughts:

Writing meta.yaml's manually is a chore, anything that reduces toil (thereby lowering barrier to entry for Pangeo Forge) is a good idea, assuming it can be implemented in a manageable fashion. So 👍 on the general concept.
I think it will be difficult to reliably generate meta.yamls without a schema to validate against. We have a relatively complete draft of that, but I think it should be moved into pangeo-forge-runner (and possibly programmatically employed there, to validate feedstock contents): https://github.com/pangeo-forge/pangeo-forge-runner/issues/93
Design-wise, I think of the relationship between core components of Pangeo Forge as:
```
flowchart TD
    pangeo-forge-runner-->meta.yaml
    meta.yaml-->recipe.py
```
Where recipe.pys (i.e. the pangeo-forge-recipes layer) can function (and therefore be developed and tested) independent from meta.yamls and/or pangeo-forge-runner, because they are the lowest level of the stack.
For both testing and UI purposes, I think it's important that code that modifies/extends recipe.pys is factored separately from any code that modifies/extends/generates meta.yamls. Therefore, I would argue that a single PangeoForgeCMR object should not both generate FilePatterns and also meta.yamls. On the testing side, a core design principle of Pangeo Forge is that pangeo-forge-runner and pangeo-forge-recipes are not released in synchronized fashion, which is achieved by keeping -runner's API only loosely coupled to that of -recipes. A monolithic PangeoForgeCMR object here would cut against that design and introduce complex dependencies for testing. More on the UI aspect in the next bullet.
From a UI standpoint, the question that comes to mind is when and how is code that auto-generates a meta.yaml run? To me what comes to mind is that this falls nicely under the umbrella of an (as yet undiscussed & unimplemented) pangeo-forge-runner init command, which could have the function of (optionally) generating a new project structure in a directory. This feels consistent with the UI of other familiar tooling, e.g. npm init, which takes an optional <initializer> argument, for the type of project to generate. So we could envision:
```
pangeo-forge-runner init <initializer>
```
being added to pangeo-forge-runner, and then
```
pangeo-forge-runner init cmr --short-name='MOD02QKM' --version='6.1'
```
could be supported by a create-cmr-feedstock initializer, which plays the role (in the npm init analogy) of something like create-react-app. Importantly, the initializer is a different package from the tooling it does the templating for (final reference to the npm example, I promise)... just as create-react-app is a different package from react itself.

In this paradigm, create-cmr-feedstock initializer could be pretty opinionated, for generating:

# meta.yaml
provenance:
  ...
  source_data:
    cmr:
      short_name: GPM_3IMERGDL
      version: '06'

and

from pangeo_forge_cmr import files_from_cmr

file_pattern = files_from_cmr(
    short_name='GPM_3IMERGDL',
    version='06',
    nitems_per_file=1,
    concat_dim='time',
)

With really the only difference from the initial proposal being the layer that this is generation is not handled in pangeo_forge_cmr (which extends -recipes) and rather in a separate (higher) create-cmr-feedstock initializer layer (which extends -runner).

Thoughts? xref https://github.com/pangeo-forge/pangeo-forge-runner/issues/94

yuvipanda / pangeo-forge-cmr

Retrieve provenance information from CMR #4