Production re-ID for SPRUCE (`gold:Gs0110138`/`nmdc:sty-11-33fbta56`) and Stegen (`gold:Gs0114663`/`nmdc:sty-11-aygzgv51`)

[aclum]

Deliverable this task is associated with

_See Deliverables tab here: https://docs.google.com/spreadsheets/d/1lLQ3wAwJzvxujER6-b4iDnnHx06MYsDbmIBMnu1IOeI/edit#gid=0

• 2,6

RACI

Tag people in their roles

• Responsible: @mbthornton-lbl @eecavanna
• Accountable: @aclum @scanon @corilo Paul
• Consulted: @shreddd napa squad
• Informed: napa squad, leadership

Describe the task

• [x] 1. Prepare the production environment (Eric)

  1. Take production Runtime (and friends) offline
     1. On Rancher, in the nmdc namespace, scale the following workloads down to 0 replicas:
        • aggregations (1 → 0)
        • scheduler (1 → 0)
        • dagster-daemon (1 → 0)
        • runtime-api (3 → 0)

  2. Limit production Mongo access

     1. At a mongosh shell, use the root user to connect to the production Mongo server and temporarily modify roles:

        You can get the root Mongo user's password from Rancher.

        
        use admin;

     // Revoke "read" access from team members (they already lack write access). db.getRole("nmdc_reader", { showPrivileges: true }); db.updateRole("nmdc_reader", { privileges: [], roles: [] });

     // Revoke "read" and "write" access from the aggregator (normally, it can write to some collections). db.getRole("nmdc_aggregator", { showPrivileges: true }); db.updateRole("nmdc_aggregator", { privileges: [], roles: [] });

     // Revoke "read" and "write" access from the scheduler (normally, it can write to the jobs collection). db.getRole("nmdc_scheduler", { showPrivileges: true }); db.updateRole("nmdc_scheduler", { privileges: [], roles: [] });

• [x] 2. Back up the production Mongo databases (Eric)
  1. Back up the production Mongo databases
     1. On Rancher, go to "Workloads" > "CronJobs", and — in the nmdc namespace — locate the CronJob named backup-mongo
     2. Click the 3-dot menu on that row and select Run Now
  2. Confirm the databases were backed up
     1. On dtn01.nersc.gov, confirm the backup exists

        # List the 5 most recent dumps.
        cd /global/cfs/cdirs/m3408/nmdc-mongodumps/
        ls -ltr | tail -5

• [x] 3. Prepare the Napa environment (Eric)

  1. Take Napa Runtime (and friends) offline
     1. On Rancher, in the nmdc-napa namespace, scale the following workloads down to 0 replicas:
        • scheduler (1 → 0)
        • dagster-daemon (1 → 0)
        • runtime-api (3 → 0)

  2. Limit Napa Mongo access

     1. At a mongosh shell, use the root user to connect to the Napa Mongo server and temporarily modify roles:

        You can get the root Mongo user's password from Rancher.

        
        use admin;

     // Revoke "write" access from re-ID squad members. // Note: We keep their "read" access so they can help "spot check" the re-ID-ing. // Note: Squad members actively performing the re-ID-ing will be given // "write" access via a different role later. db.getRole("nmdc_editor", { showPrivileges: true }); db.updateRole("nmdc_editor", { privileges: [], roles: [ { db: "nmdc", role: "read" }, { db: "nmdc_updated", role: "read" }, { db: "nmdc_deleted", role: "read" }, { db: "nmdc_changesheet_submission_results", role: "read" } ]});

     // Revoke "read" and "write" access from the aggregator (normally, it can write to some collections). db.getRole("nmdc_aggregator", { showPrivileges: true }); db.updateRole("nmdc_aggregator", { privileges: [], roles: [] });

     // Revoke "read" and "write" access from the scheduler (normally, it can write to the jobs collection). db.getRole("nmdc_scheduler", { showPrivileges: true }); db.updateRole("nmdc_scheduler", { privileges: [], roles: [] });

  3. Back up the Napa databases
     1. On Rancher, go to "Workloads" > "CronJobs", and — in the nmdc-napa namespace — locate the CronJob named backup-mongo
     2. Click the 3-dot menu on that row and select Run Now
  4. Confirm the databases were backed up
     1. On dtn01.nersc.gov, confirm the backup exists

        # List the 5 most recent dumps.
        cd /global/cfs/cdirs/m3408/nmdc-mongodumps/
        ls -ltr | tail -5

  5. Drop the Napa databases

     1. On dtn01.nersc.gov, use mongosh to connect to the Napa Mongo server and drop the databases:

        
        cd /global/cfs/projectdirs/m3408/nmdc-mongodumps

     ./bin/mongosh \ --host="mongo-loadbalancer.nmdc-napa.production.svc.spin.nersc.org" \ --username "root" \ --eval 'use nmdc' --eval 'db.dropDatabase()' \ --eval 'use nmdc_updated' --eval 'db.dropDatabase()' \ --eval 'use nmdc_deleted' --eval 'db.dropDatabase()' \ --eval 'use nmdc_changesheet_submission_results' --eval 'db.dropDatabase()'

  6. Restore the production dump to the Napa Mongo server

     1. On dtn01.nersc.gov, use mongorestore to restore the production dump you created earlier, into the Napa Mongo server.

        Replace the {__TIMESTAMP__} placeholder below with the real ending of the dump's directory name.

        
        cd /global/cfs/projectdirs/m3408/nmdc-mongodumps

     ./bin/mongorestore \ --host="mongo-loadbalancer.nmdc-napa.production.svc.spin.nersc.org" \ --username="root" \ --verbose --stopOnError --drop --preserveUUID \ --nsInclude 'nmdc.' \ --nsInclude 'nmdc_deleted.' \ --nsInclude 'nmdc_updated.' \ --nsInclude 'nmdc_changesheet_submission_results.' \ --dir="./dumpnmdc-prod{TIMESTAMP}"

     
     > This typically takes 7-8 minutes.

  7. Grant read/write access to the team members actively performing re-ID-ing (i.e. Michael and Alicia)

     1. At a mongosh shell, use the root user to connect to the Napa Mongo server and temporarily assign roles:

        
        use admin;

     // Temporarily assign them the "nmdc_runtime" role, which has "read/write" access to all "nmdc*" collections. db.grantRolesToUser('alicia.clum-napa', ['nmdc_runtime']); db.grantRolesToUser('michael.thornton-napa', ['nmdc_runtime']);

  8. Bring the Napa Runtime (and friends) back online
     1. On Rancher, in the nmdc-napa namespace, scale the following workloads up:
        • runtime-api (0 → 3)
        • dagster-daemon (0 → 1)
        • scheduler (0 → 1)
• [x] 4. https://github.com/microbiomedata/nmdc_automation/issues/167
• [x] 5. https://github.com/microbiomedata/nmdc_automation/issues/170
• [x] 6. microbiomedata/nmdc_automation#172 re-id study_set,biosample_set,and omics_processing_set records (Micheal)
• [x] 7. delete straggler legacy data objects
  1. delete_bioscales_data_objects.js
  2. re_id_tool.py delete-old-binning-data
• [x] 8. run pure-export and validate copy with linkml-validate (Alicia/Michael)
  1. run make all to generate materialized version of the schema
  2. run linkml-validate using -s project/nmdc_materialized_patterns.yaml
• [x] 9. run script to validate that all id slots are napa complaint based on structured pattern.syntax for that Class as well as all references to nmdc identifiers (part_of, was_informed_by, has_input, has_output, was_generated_by, metagenome_annotation_id, metaproteomic_analysis_id) (Alicia/Michael)
• [x] 10. #717

• [x] 11. (was 10.) Dump Napa Mongo and restore to production Mongo server (Eric)

  1. Revoke Napa Mongo write access from the team members actively performing re-ID-ing (i.e. Michael and Alicia)

     1. At a mongosh shell, use the root user to connect to the Napa Mongo server and modify roles:

        
        use admin;

     // Revoke their "write" access to all "nmdc*" collections. // Note: They'll still have "read" access because they'll still have the nmdc_editor role, // which currently (since we constrained it earlier) provides only "read" access. db.revokeRolesFromUser('alicia.clum-napa', ['nmdc_runtime']); db.revokeRolesFromUser('michael.thornton-napa', ['nmdc_runtime']);

  2. Take the Napa Runtime (and friends) offline
     1. On Rancher, in the nmdc-napa namespace, scale the following workloads down:
        • scheduler (1 → 0)
        • dagster-daemon (1 → 0)
        • runtime-api (3 → 0)
  3. Back up the Napa databases (these contained the re-ID-ed data)
     1. On Rancher, go to "Workloads" > "CronJobs", and — in the nmdc-napa namespace — locate the CronJob named backup-mongo
     2. Click the 3-dot menu on that row and select Run Now
  4. Confirm the databases were backed up
     1. On dtn01.nersc.gov, confirm the backup exists

        # List the 5 most recent dumps.
        cd /global/cfs/cdirs/m3408/nmdc-mongodumps/
        ls -ltr | tail -5

  5. Drop the production databases

     1. On dtn01.nersc.gov, use mongosh to connect to the production Mongo server and drop the databases:

        
        cd /global/cfs/projectdirs/m3408/nmdc-mongodumps

     ./bin/mongosh \ --host="mongo-loadbalancer.nmdc.production.svc.spin.nersc.org" \ --username "root" \ --eval 'use nmdc' --eval 'db.dropDatabase()' \ --eval 'use nmdc_updated' --eval 'db.dropDatabase()' \ --eval 'use nmdc_deleted' --eval 'db.dropDatabase()' \ --eval 'use nmdc_changesheet_submission_results' --eval 'db.dropDatabase()'

  6. Restore the Napa dump (containing the re-ID-ed data) to the production Mongo server

     1. On dtn01.nersc.gov, use mongorestore to restore the Napa dump you created most recently, into the production Mongo server.

        Replace the {__TIMESTAMP__} placeholder below with the real ending of the dump's directory name.

        
        cd /global/cfs/projectdirs/m3408/nmdc-mongodumps

     ./bin/mongorestore \ --host="mongo-loadbalancer.nmdc.production.svc.spin.nersc.org" \ --username="root" \ --verbose --stopOnError --drop --preserveUUID \ --nsInclude 'nmdc.' \ --nsInclude 'nmdc_deleted.' \ --nsInclude 'nmdc_updated.' \ --nsInclude 'nmdc_changesheet_submission_results.' \ --dir="./dumpnmdc-napa{TIMESTAMP}"

     
     > This typically takes 7-8 minutes.

• [x] 12. Restore the production environment and access to it (Eric)

  1. Restore production Mongo access

     1. At a mongosh shell, use the root user to connect to the production Mongo server and restore roles to their original definitions

        
        use admin;

     // Allow team members and the aggregator to once again "read" from the nmdc* databases. db.getRole("nmdc_reader", { showPrivileges: true }); db.updateRole("nmdc_reader", { privileges: [], roles: [ { db: "nmdc", role: "read" }, { db: "nmdc_updated", role: "read" }, { db: "nmdc_deleted", role: "read" }, { db: "nmdc_changesheet_submission_results", role: "read" } ]});

     // Allow the aggregator to once again "read" from the nmdc database and "write" to some collections. db.getRole("nmdc_aggregator", { showPrivileges: true }); db.updateRole("nmdc_aggregator", { privileges: [ { resource: { db: "nmdc", collection: "metap_gene_function_aggregation" }, actions: ["find", "insert", "update", "remove"] }, { resource: { db: "nmdc", collection: "functional_annotation_agg" }, actions: ["find", "insert", "update", "remove"] }, ], roles: [ { db: "nmdc", role: "read" }, ], });

     // Allow the scheduler to once again "read" from the nmdc database and "write" to the jobs collection. db.getRole("nmdc_scheduler", { showPrivileges: true }); db.updateRole("nmdc_scheduler", { privileges: [ { resource: { db: "nmdc", collection: "jobs" }, actions: ["find", "insert", "update", "remove"] } ], roles: [ { db: "nmdc", role: "read" }, ]
     });

  2. Bring the production Runtime (and friends) back online
     1. On Rancher, in the nmdc namespace, scale the following workloads up:
        • runtime-api (0 → 3)
        • dagster-daemon (0 → 1)
        • scheduler (0 → 1)
        • aggregations (0 → 1)
• [x] 13. ingest from mongo prod -> local portal instance (Mike/Mary)
• [x] 14. ingest from mong prod -> data portal prod (Mike/Mary/automatic)
• [x] 15. review of data portal (Alicia) - all studies have nmdc identifiers and have the correct number of samples and omics records per type.

Criteria for completion

• All nmdc identifiers (slot id for Classes Biosample, ProcessedSample, classes that inherit from WorkflowExecutionActivity, DataObject; slot metagenome_annotation_id for FunctionalAnnotationAggMember; slot metaproteomic_analysis_id for metap_gene_function_aggregation)

Estimate people time

• 3 person, 8 hours

Completion Date (Goal)

• May 29

Target Sprint Start & End Dates

• Start: May 20
• End: May 31

Tag Blocker/Contingent upon issues https://github.com/microbiomedata/nmdc_automation/issues/178

[eecavanna]

Outstanding preparation tasks for @eecavanna:

• [x] Update instructions to account for existence of the newly-defined "standard" Mongo roles (e.g. use nmdc_editor instead of nmdc_reader_writer). This is a < 10 minute task.

[mbthornton-lbl]

29052024_nmdc_napa_linkml_validate.log

[mbthornton-lbl]

Summary of findings from linkml-validate vs. the materialized patterns schema:

• 100 data_objects with non-conforming IDs e.g. emsl:output_451859
• 100 nom_analysis_activity with non-conforming IDs e.g. nmdc:wfnom-11-1s9c9921
• 6 read_based_taxonomy_activity with non-conforming IDs e.g. nmdc:wfrbt-11-0227se86.1.1
• 12 read_qc_analysis_activity with non-conforming IDs e.g. nmdc:wfrqc-11-1g93yh86.1.1

[aclum]

Orphaned data_object_set and functional_annotation_agg will be dealt with separately (https://github.com/microbiomedata/nmdc-schema/issues/2011), these are not visible on the data portal. Workflow identifier issues (no id version and multiple ID versions) are being handled separately (https://github.com/microbiomedata/nmdc-schema/issues/2010 and https://github.com/microbiomedata/nmdc_automation/issues/186)