Use NEON soil data to establish data sharing process with IMG

[aclum]

NMDC roadmap this task is associated with 1.5 Establish data sharing process with IMG/M to support NMDC workfow outputs and associated links

RACI

Tag people in their roles

• Responsible: @aclum, @mhuntemann, Reddy
• Accountable: @aclum, Amy C.
• Consulted: Amy C.
• Informed: @emileyfadrosh

Describe the the task

• [ ] pilot IMG ingest using 2 projects already in GOLD, ingest should include annotation, bins from metabat2, and reads based analysis
• [x] Reddy to develop a mechanism to create new GOLD biosamples from NMDC metadata (currently only 1/3 of soil projects are already in GOLD
• [ ] IMG to ingest the remaining workflow data (~1,500 annotations w/87 of them being re-annotations of the JGI data so the entire project uses annotation 5.2)

Criteria for completion

• ~1500 annotations & binning results ingested into IMG from NMDC

Estimate people time

• ~40+ hours split b/w Marcel, Alicia, Reddy, Amy

Completion Date (Goal)

• Sept 21 for JGI IMG director's update for pilot work
• Oct 20 for the rest of the soil results

Target Sprint Start & End Dates

• Start: Sept 11
• End: Oct 20

Tag Blocker/Contingent upon issues

[aclum]

Two results were ingested but this was only annotation and reads based analysis. IMG needs to test ingesting binning results and IMG needs to ingest the remainder of the records. Reddy did test generating new biosamples for projects that weren't in GOLD.

[aclum]

I gave Amy the mapping for the rest of the soil samples, the GOLD team is working on curating the biosamples that didn't exist in GOLD previously. Moving to the next sprint since this won't be finished by Friday.

[ssarrafan]

Removing this from the sprint and adding the backlog label. Not updates in 2 weeks on the ticket.

[aclum]

Marcel imported ~50 projects, about 10% had errors b/c of nmdc run failures. @Michal-Babins is looking into the failures.

[ssarrafan]

@Michal-Babins @aclum is this still in progress?

[aclum]

Michal restarted the failures and some of them are still running. Additionally, Reddy & the GOLD team finished curating the rest of the biosamples so there are another ~1000 annotations that IMG can now import. Leaving in the backlog for now since most of the work is on the jgi side.

[aclum]

Marcel has been ingesting more data this week.

[aclum]

Patrick ran into issues on the projects we re-annotated started from JGI assemblies. There are a few tickets that I'll link that need to be resolved. blocked on https://github.com/microbiomedata/nmdc_automation/issues/35 and https://github.com/microbiomedata/mg_annotation/issues/24

[aclum]

This has been blocked on a variety of issues with the binning, schema, and the automation code. We expect to have some binning results this quarter which could be passed off to JGI.