Open doug-newman-nasa opened 1 year ago
Thanks for opening this up! /cc @cisaacstern to see what he thinks. I'll try to come back and provide a more whole response in a day or so!
Thanks for thinking through this @doug-newman-nasa, and for the tag @yuvipanda. Some thoughts:
meta.yaml
s without a schema to validate against. We have a relatively complete draft of that, but I think it should be moved into pangeo-forge-runner
(and possibly programmatically employed there, to validate feedstock contents): https://github.com/pangeo-forge/pangeo-forge-runner/issues/93Design-wise, I think of the relationship between core components of Pangeo Forge as:
flowchart TD
pangeo-forge-runner-->meta.yaml
meta.yaml-->recipe.py
Where recipe.py
s (i.e. the pangeo-forge-recipes
layer) can function (and therefore be developed and tested) independent from meta.yaml
s and/or pangeo-forge-runner
, because they are the lowest level of the stack.
recipe.py
s is factored separately from any code that modifies/extends/generates meta.yaml
s. Therefore, I would argue that a single PangeoForgeCMR
object should not both generate FilePattern
s and also meta.yaml
s. On the testing side, a core design principle of Pangeo Forge is that pangeo-forge-runner
and pangeo-forge-recipes
are not released in synchronized fashion, which is achieved by keeping -runner
's API only loosely coupled to that of -recipes
. A monolithic PangeoForgeCMR
object here would cut against that design and introduce complex dependencies for testing. More on the UI aspect in the next bullet.From a UI standpoint, the question that comes to mind is when and how is code that auto-generates a meta.yaml
run? To me what comes to mind is that this falls nicely under the umbrella of an (as yet undiscussed & unimplemented) pangeo-forge-runner init
command, which could have the function of (optionally) generating a new project structure in a directory. This feels consistent with the UI of other familiar tooling, e.g. npm init, which takes an optional <initializer>
argument, for the type of project to generate. So we could envision:
pangeo-forge-runner init <initializer>
being added to pangeo-forge-runner
, and then
pangeo-forge-runner init cmr --short-name='MOD02QKM' --version='6.1'
could be supported by a create-cmr-feedstock
initializer, which plays the role (in the npm init
analogy) of something like create-react-app
. Importantly, the initializer is a different package from the tooling it does the templating for (final reference to the npm
example, I promise)... just as create-react-app
is a different package from react
itself.
In this paradigm, create-cmr-feedstock
initializer could be pretty opinionated, for generating:
# meta.yaml
provenance:
...
source_data:
cmr:
short_name: GPM_3IMERGDL
version: '06'
and
from pangeo_forge_cmr import files_from_cmr
file_pattern = files_from_cmr(
short_name='GPM_3IMERGDL',
version='06',
nitems_per_file=1,
concat_dim='time',
)
With really the only difference from the initial proposal being the layer that this is generation is not handled in pangeo_forge_cmr
(which extends -recipes
) and rather in a separate (higher) create-cmr-feedstock
initializer layer (which extends -runner
).
Thoughts? xref https://github.com/pangeo-forge/pangeo-forge-runner/issues/94
Given the above, I'm leaning to an implementation where we have a class constructor that figures out the input collection and/or retrieves the provenance information for a collection and populates the yaml file. Then users invoke the get files method. For example:
Alternatively, if the short name and version id are to be extracted from meta.yaml
Where the yaml file contains,
Please give your thoughts on this API suggestion before I implement!