reprocess datasets through MAG workflow

[aclum]

RACI

Tag people in their roles

• Responsible: @Michal-Babins
• Accountable: @hubin-keio
• Consulted: @chienchi
• Informed: @aclum

Describe the the task

• [ ] reprocess workflow runs since July 2023 (NEON, TRiP) due to error in binning workflow which specified incorrect path to checkM reference data

Criteria for completion

• new workflow results with .2 identifiers should be in mongo prod with a processed data greater than Feb 20, 2024.

Estimate people time

• [Hours or days of people time. 1 person, 4 hours]

Completion Date (Goal)

• [GOAL!]

Target Sprint Start & End Dates

• Start:
• End:

Tag Blocker/Contingent upon issues

• @chienchi to confirm end to end workflow test completes for a dataset where we expect bins to be generated.

[aclum]

related to https://github.com/microbiomedata/metaMAGs/issues/19

[Michal-Babins]

I am gathering the neccessary information to reprocess these. I will have @scanon review this. Once the mags chances are ready, we can resubmit trip and neon soil for mags only with the bumped version. We can start with TriP first since it's smaller and move to NEON.

[Michal-Babins]

I am running a few more tests to make sure the runtime attributes are set correctly with the updates, the changes @chienchi made look good, but need to confirm that the runtime requirements are stable. I had one project crash on me due to OOM and the requested memory was too low.

[aclum]

We need to add the new file types including capturing the low quality bins, see https://github.com/microbiomedata/nmdc-schema/pull/1791. I'm working on getting runtime dev updated.

[ssarrafan]

@Michal-Babins @aclum what is the status of this issue?

[aclum]

Michal was testing the runtime requirements and we need an updated runtime prod env to be able to handle the new file type numerations. This first needs a new update to runtime dev. We updated runtime dev to 10.1 but that release is incomplete per Patrick C. so we need runtime dev to be updated to 10.1.1, then test, then runtime prod needs to be updated to 10.1.1. cc @kaijli

[ssarrafan]

Michal was testing the runtime requirements and we need an updated runtime prod env to be able to handle the new file type numerations. This first needs a new update to runtime dev. We updated runtime dev to 10.1 but that release is incomplete per Patrick C. so we need runtime dev to be updated to 10.1.1, then test, then runtime prod needs to be updated to 10.1.1. cc @kaijli

Thanks Alicia. Should this be moved to the next sprint?

[ssarrafan]

Removing from sprint. @aclum @kaijli let me know if this is still active.

[aclum]

We need to resolve the blockers first, adding backlog label so we don't lose this.

[aclum]

@aclum to add study IDs.

[aclum]

Provided list of 1970 MAGs activities that need to be reprocessed via slack. Search by version is below, this returned the same number of records as a search by date ({'started_at_time':{ $gte : "2023-07-01"}})

db.getCollection('mags_activity_set').find({
--
version: 'v1.0.6'
});

Studies impacted (TRiP nmdc:sty-11-hdd4bf83, NEON soil nmdc:sty-11-34xj1150, NEON benthic nmdc:sty-11-pzmd0x14)

[aclum]

nmdc.mags_activity_set.csv

[aclum]

Reattaching the list that I provided in slack on July 28th 2023 to Shane and Michal. These are projects where we know the JGI workflow produced bins with a mapping to NMDC's corresponding omics ID.

IMG (15).xlsx

cc @scanon

[aclum]

Moving this to blocked as this is on hold until we get the eukcc component code from Neha.