ingest GLBRC submission to mongo / DP

[mslarae13]

Ingest the GLBRC metadata submission on NMDC submission portal to mongo and data portal

NMDC submission portal submission ID: d32b5eb6-71e8-4c98-8a57-3e64d05718b4

Notes about this submission

• Skip depth. This isn't required for plants, but the submission portal says it is
• Skip chem_administration this is for specific chemicals & all information is also captured in fertilizer_regmn
• need to ingest the metadata that is in the NMDC submission portal & additional metadata where slots weren't available in the plant interface
• the DOI provided in the submission (10.46936/10.25585/60000818) is an award DOI

study_category = research_study

funding statement = Support for this research was provided by the Great Lakes Bioenergy Research Center, U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research (Awards DE-SC0018409 and DE-FC02-07ER64494), by the National Science Foundation Long-term Ecological Research Program (DEB 1832042) at the Kellogg Biological Station, and by Michigan State University AgBioResearch.

PI image can be found

All metadata is available Ingested_2023-11_nmdc_sample_metadata.xlsx (This has what is in the submission portal & the metadata that was not in the submission)

GLBRC metadata terms / column headers are mapped to NMDC slots here

TBD

• [ ] Reassign some metadata fields to the site level
• [ ] Some metadata is soil data. Need to develop associating another samples biogeochem data as metadata for a plant
• [ ] Phenology metadata is at the site / plot level and is a general assessment.
• [ ] Are GLBRC columns / metadata terms broadly used? Is it worth adding aliases or mappings to NMDC schema?

[mslarae13]

@cmungall I added mappings to the term map file linked above. (GLBRC metadata terms / column headers are mapped to NMDC slots here) ... can you look at the mapping levels I selected. I've never done this & want to be sure I am doing it right.

[mslarae13]

@pkalita-lbl you should have everything you need now. Let me know. if you don't!

[ssarrafan]

@pkalita-lbl what's the status of this issue? Active? Backlog?

[pkalita-lbl]

End of sprint update:

• PR with supporting code changes is open and waiting for review: https://github.com/microbiomedata/nmdc-runtime/pull/383
• Waiting for Adina to clarify units of absolute humidity (AH) as either g/m3 or kPa (this is driven by external configuration -- not code -- so this doesn't need to hold up the PR)

Moving to next sprint.

[mslarae13]

Adina clarified units. And I updated a unit to match UCUM.

[ssarrafan]

Adina clarified units. And I updated a unit to match UCUM.

Thanks @mslarae13 FYI @pkalita-lbl

[pkalita-lbl]

So I think where we're at now:

• Runtime code changes are in.
• I believe all the updates to the metadata in the submission portal has been done (getting the location format right, using UCUM Cel instead of Celsius).
• The files in https://portal.nersc.gov/project/m3408/submission_data/d32b5eb6-71e8-4c98-8a57-3e64d05718b4/ are going to be the official source of extra biosample metadata (including units via the mapping file).

Next steps:

• One of the units they're using (they report it as "microEinsteinPerMeterSquaredPerSecond") is not UCUM-compatible (see also). I will leaf through the UCUM spec a bit more to figure out of there is a standard way of representing non-standard things.
• Run the pipeline on dev & do one last sanity check.
• Run the pipeline on prod.

[mslarae13]

standard way of representing non-standard things. 🙃

Thank @pkalita-lbl

[pkalita-lbl]

Updates:

• Regarding the UCUM-incompatible unit "microEinsteinPerMeterSquaredPerSecond" I went ahead an used u[E]/m2/s. This is not strictly UCUM-compliant, but there is no UCUM-compliant way to express that unit. UCUM uses square brackets to denote what they call "arbitrary units" (generally non-SI stuff), but the UCUM spec still enumerates the allowed arbitrary units (Einstein is not one of them). The square brackets are also sort of a warning sign that the unit cannot be automatically converted to another unit. So lacking any better options I went with [E] to represent the Einstein because it's what UCUM would probably include if it included the Einstein (see issue linked in previous comment).
• I ran the pipeline to bring the study into dev Mongo and prod Mongo. Here it is on prod: https://api.microbiomedata.org/studies/nmdc%3Asty-11-e4yb9z58
• It will be visible on the data portal after the next ingest runs, which will be after nmdc-server changes have been made to update the ingest code to accommodate Study class changes. Kitware is working on that.

[pkalita-lbl]

I'm going to close this issue now. There's nothing else specific to the GLBRC submission that needs to be done. That data is in Mongo on dev and prod. It will show up on the data portal in with the next ingest, which will happen once the changes in https://github.com/microbiomedata/nmdc-server/pull/1084 propagate through.