Closed aclum closed 3 months ago
Outstanding preparation tasks for @eecavanna:
nmdc_editor
instead of nmdc_reader_writer
). This is a < 10 minute task.Summary of findings from linkml-validate
vs. the materialized patterns schema:
emsl:output_451859
nmdc:wfnom-11-1s9c9921
nmdc:wfrbt-11-0227se86.1.1
nmdc:wfrqc-11-1g93yh86.1.1
Orphaned data_object_set and functional_annotation_agg will be dealt with separately (https://github.com/microbiomedata/nmdc-schema/issues/2011), these are not visible on the data portal. Workflow identifier issues (no id version and multiple ID versions) are being handled separately (https://github.com/microbiomedata/nmdc-schema/issues/2010 and https://github.com/microbiomedata/nmdc_automation/issues/186)
Deliverable this task is associated with
_See Deliverables tab here: https://docs.google.com/spreadsheets/d/1lLQ3wAwJzvxujER6-b4iDnnHx06MYsDbmIBMnu1IOeI/edit#gid=0
RACI
Tag people in their roles
Describe the task
[x] 1. Prepare the production environment (Eric)
nmdc
namespace, scale the following workloads down to 0 replicas:aggregations
(1 → 0)scheduler
(1 → 0)dagster-daemon
(1 → 0)runtime-api
(3 → 0)Limit production Mongo access
mongosh
shell, use theroot
user to connect to the production Mongo server and temporarily modify roles:// Revoke "read" access from team members (they already lack write access). db.getRole("nmdc_reader", { showPrivileges: true }); db.updateRole("nmdc_reader", { privileges: [], roles: [] });
// Revoke "read" and "write" access from the aggregator (normally, it can write to some collections). db.getRole("nmdc_aggregator", { showPrivileges: true }); db.updateRole("nmdc_aggregator", { privileges: [], roles: [] });
// Revoke "read" and "write" access from the scheduler (normally, it can write to the
jobs
collection). db.getRole("nmdc_scheduler", { showPrivileges: true }); db.updateRole("nmdc_scheduler", { privileges: [], roles: [] });nmdc
namespace — locate the CronJob namedbackup-mongo
Run Now
dtn01.nersc.gov
, confirm the backup exists[x] 3. Prepare the Napa environment (Eric)
nmdc-napa
namespace, scale the following workloads down to 0 replicas:scheduler
(1 → 0)dagster-daemon
(1 → 0)runtime-api
(3 → 0)Limit Napa Mongo access
mongosh
shell, use theroot
user to connect to the Napa Mongo server and temporarily modify roles:// Revoke "write" access from re-ID squad members. // Note: We keep their "read" access so they can help "spot check" the re-ID-ing. // Note: Squad members actively performing the re-ID-ing will be given // "write" access via a different role later. db.getRole("nmdc_editor", { showPrivileges: true }); db.updateRole("nmdc_editor", { privileges: [], roles: [ { db: "nmdc", role: "read" }, { db: "nmdc_updated", role: "read" }, { db: "nmdc_deleted", role: "read" }, { db: "nmdc_changesheet_submission_results", role: "read" } ]});
// Revoke "read" and "write" access from the aggregator (normally, it can write to some collections). db.getRole("nmdc_aggregator", { showPrivileges: true }); db.updateRole("nmdc_aggregator", { privileges: [], roles: [] });
// Revoke "read" and "write" access from the scheduler (normally, it can write to the
jobs
collection). db.getRole("nmdc_scheduler", { showPrivileges: true }); db.updateRole("nmdc_scheduler", { privileges: [], roles: [] });nmdc-napa
namespace — locate the CronJob namedbackup-mongo
Run Now
dtn01.nersc.gov
, confirm the backup existsDrop the Napa databases
dtn01.nersc.gov
, usemongosh
to connect to the Napa Mongo server and drop the databases:./bin/mongosh \ --host="mongo-loadbalancer.nmdc-napa.production.svc.spin.nersc.org" \ --username "root" \ --eval 'use nmdc' --eval 'db.dropDatabase()' \ --eval 'use nmdc_updated' --eval 'db.dropDatabase()' \ --eval 'use nmdc_deleted' --eval 'db.dropDatabase()' \ --eval 'use nmdc_changesheet_submission_results' --eval 'db.dropDatabase()'
Restore the production dump to the Napa Mongo server
dtn01.nersc.gov
, usemongorestore
to restore the production dump you created earlier, into the Napa Mongo server../bin/mongorestore \ --host="mongo-loadbalancer.nmdc-napa.production.svc.spin.nersc.org" \ --username="root" \ --verbose --stopOnError --drop --preserveUUID \ --nsInclude 'nmdc.' \ --nsInclude 'nmdc_deleted.' \ --nsInclude 'nmdc_updated.' \ --nsInclude 'nmdc_changesheet_submission_results.' \ --dir="./dumpnmdc-prod{TIMESTAMP}"
Grant read/write access to the team members actively performing re-ID-ing (i.e. Michael and Alicia)
mongosh
shell, use theroot
user to connect to the Napa Mongo server and temporarily assign roles:// Temporarily assign them the "nmdc_runtime" role, which has "read/write" access to all "nmdc*" collections. db.grantRolesToUser('alicia.clum-napa', ['nmdc_runtime']); db.grantRolesToUser('michael.thornton-napa', ['nmdc_runtime']);
nmdc-napa
namespace, scale the following workloads up:runtime-api
(0 → 3)dagster-daemon
(0 → 1)scheduler
(0 → 1)delete_bioscales_data_objects.js
re_id_tool.py delete-old-binning-data
make all
to generate materialized version of the schemalinkml-validate
using-s project/nmdc_materialized_patterns.yaml
[x] 11. (was 10.) Dump Napa Mongo and restore to production Mongo server (Eric)
Revoke Napa Mongo write access from the team members actively performing re-ID-ing (i.e. Michael and Alicia)
mongosh
shell, use theroot
user to connect to the Napa Mongo server and modify roles:// Revoke their "write" access to all "nmdc*" collections. // Note: They'll still have "read" access because they'll still have the
nmdc_editor
role, // which currently (since we constrained it earlier) provides only "read" access. db.revokeRolesFromUser('alicia.clum-napa', ['nmdc_runtime']); db.revokeRolesFromUser('michael.thornton-napa', ['nmdc_runtime']);scheduler
(1 → 0)dagster-daemon
(1 → 0)runtime-api
(3 → 0)nmdc-napa
namespace — locate the CronJob namedbackup-mongo
Run Now
dtn01.nersc.gov
, confirm the backup existsDrop the production databases
dtn01.nersc.gov
, usemongosh
to connect to the production Mongo server and drop the databases:./bin/mongosh \ --host="mongo-loadbalancer.nmdc.production.svc.spin.nersc.org" \ --username "root" \ --eval 'use nmdc' --eval 'db.dropDatabase()' \ --eval 'use nmdc_updated' --eval 'db.dropDatabase()' \ --eval 'use nmdc_deleted' --eval 'db.dropDatabase()' \ --eval 'use nmdc_changesheet_submission_results' --eval 'db.dropDatabase()'
Restore the Napa dump (containing the re-ID-ed data) to the production Mongo server
dtn01.nersc.gov
, usemongorestore
to restore the Napa dump you created most recently, into the production Mongo server../bin/mongorestore \ --host="mongo-loadbalancer.nmdc.production.svc.spin.nersc.org" \ --username="root" \ --verbose --stopOnError --drop --preserveUUID \ --nsInclude 'nmdc.' \ --nsInclude 'nmdc_deleted.' \ --nsInclude 'nmdc_updated.' \ --nsInclude 'nmdc_changesheet_submission_results.' \ --dir="./dumpnmdc-napa{TIMESTAMP}"
[x] 12. Restore the production environment and access to it (Eric)
Restore production Mongo access
mongosh
shell, use theroot
user to connect to the production Mongo server and restore roles to their original definitions// Allow team members and the aggregator to once again "read" from the
nmdc*
databases. db.getRole("nmdc_reader", { showPrivileges: true }); db.updateRole("nmdc_reader", { privileges: [], roles: [ { db: "nmdc", role: "read" }, { db: "nmdc_updated", role: "read" }, { db: "nmdc_deleted", role: "read" }, { db: "nmdc_changesheet_submission_results", role: "read" } ]});// Allow the aggregator to once again "read" from the
nmdc
database and "write" to some collections. db.getRole("nmdc_aggregator", { showPrivileges: true }); db.updateRole("nmdc_aggregator", { privileges: [ { resource: { db: "nmdc", collection: "metap_gene_function_aggregation" }, actions: ["find", "insert", "update", "remove"] }, { resource: { db: "nmdc", collection: "functional_annotation_agg" }, actions: ["find", "insert", "update", "remove"] }, ], roles: [ { db: "nmdc", role: "read" }, ], });// Allow the scheduler to once again "read" from the
nmdc
database and "write" to thejobs
collection. db.getRole("nmdc_scheduler", { showPrivileges: true }); db.updateRole("nmdc_scheduler", { privileges: [ { resource: { db: "nmdc", collection: "jobs" }, actions: ["find", "insert", "update", "remove"] } ], roles: [ { db: "nmdc", role: "read" }, ]});
nmdc
namespace, scale the following workloads up:runtime-api
(0 → 3)dagster-daemon
(0 → 1)scheduler
(0 → 1)aggregations
(0 → 1)Criteria for completion
Estimate people time
Completion Date (Goal)
Target Sprint Start & End Dates
Tag Blocker/Contingent upon issues https://github.com/microbiomedata/nmdc_automation/issues/178