Experiment with PO.DAAC’s cumulus-granule-to-cnm and cumulus-publish-cnm

[lisakaser]

See notes in comments as well. It may be most efficient to write the CNM files at the same time the granule metadata files are created. If we take that approach, this story can ignore the questions about cumulus-granule-to-cnm and focus on the mechanics of moving a CNM file from local infrastructure to a Cumulus Kinesis queue (the cumulus-publish-cnm part). (Probably makes sense to assume CNM files will exist and focus on the "how do we move CNM files from local infrastructure to Cumulus" question.)

Repositories of interest:

• https://github.com/podaac/cumulus-granule-to-cnm
• https://github.com/podaac/cumulus-publish-cnm

Related documentation

• NSIDC CNM structure and content guidelines
• NSIDC UMM-G structure and content guidelines

Story outcomes:

• Can we re-use the PO.DAAC lambda code to handle our CNM publishing needs, and if so, what is the level of effort required to implement a workflow in which CNM files are generated from granules living in our local infrastructure and then posted to a Cumulus Kinesis stream?
• If we can't re-use them, estimate level of effort to generate new code to support the scenario above.

[lisakaser]

Understanding is needed on how much effort it will be to modity these two PO.DAAC tools to NSIDC needs. A specific question is around posting CNM files to a Kinesis stream (posting to Cumulus) rather than on-prem location. Original project request does not require this but Ops desires it. Work effort estimate will determine if we can deliver this within this project or will need to push it to an other project or change the scope of the existing project.

[lisakaser]

This could involve a meeting with the Cumulus team that demo's PO.DAACs tools and how we currently use them, if we do.

[eigenbeam]

NSIDC forks of the two PODAAC repos:

https://github.com/nsidc/cumulus-granule-to-cnm https://github.com/nsidc/cumulus-publish-cnm

[eigenbeam]

Here are the PRs for the README updates:

https://github.com/nsidc/cumulus-granule-to-cnm/pull/1 https://github.com/nsidc/cumulus-publish-cnm/pull/1

[eigenbeam]

My assessment is that in our current architecture these two repos are not directly useful. If useful in the future, they may be used as reference, but since we are not building a Cumulus workflow these lambdas are not usable on our project.

If it is determined that it is in the scope of DUCk to publish CNM messages to a Kinesis stream, we can easily create CNM messages and post them without needing the code in these repositories.

In terms of level of effort, the work to create and publish a CNM message is in the order of a small issue/story/ticket. Perhaps 1-3 days of work.

[lisakaser]

@eigenbeam thanks for this research and assessment of work for creating and publishing CNM messages. I am closing the ticket.