E2E Testing refactored Scheduler and Watcher

[mbthornton-lbl]

Perform End-to-end testing of refactored Scheduler and Watcher in the dev environment

Deployment:

• [x] update MAGs to 1.3.12 in workflow / import configs (metagenome only)
• [x] build new image and push to https://hub.docker.com/r/microbiomedata/sched
• [x] update nmdc-dev in Spin to new image
• [x] Scheduler up and running on Spin
• [x] Watcher up and running on Perlmutter
• [x] #289
• [x] Start workers: sbatch ~/workers_perlmutter.sl

nmdcda@perlmutter:login33:~> squeue -u nmdcda
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          30091487      cron cromwell   nmdcda  R 5-10:56:57      1 login35
          30091486      cron   condor   nmdcda  R 5-10:56:55      1 login24
          33101818 regular_m nmdc_con   nmdcda PD       0:00      1 (Priority)
          33101851 regular_m nmdc_con   nmdcda PD       0:00      5 (Priority)
          33101927 regular_m nmdc_con   nmdcda PD       0:00      1 (Priority)

Integration Test Cases -

1. Scheduler creates a new unclaimed JobRequest in the jobs collection in the dev DB

Test Conditions: Create conditions for the scheduler to pickup incomplete workflow(s) and create new job requests via:

• Allow list?

• Mock Project in the dev DB?

  Pass Conditions:

• [ ] log entries showing new JobReq creation
• [ ] new unclaimed JR's in the jobs collection

2. Watcher picks up an unclaimed JobRequest and submits Cromwell job

Test Conditions: New unclaimed Job Request(s) in the dev DB (result from success in Case 1)

Pass Conditions:

• log entry showing Cromwell job submission
• correctly configured running Cromwell job

3.1 Watcher picks up successful Cromwell job and creates data directories and files

Test Conditions: Successful job on Cromwell (result from success in Case 2)

Pass Conditions:

• Watcher picks up successful cromwell job
• Watcher creates expected data directories and data files

3.2 Watcher updates NMDC dev database with successful job DataObjects and WorkflowExecution

Test Conditions: Successful job on Cromwell (result from success in Case 2)

Pass Conditions:

• log entry showing correct nmdc.Database creation from Cromwell results
• creates expected DataObject and WorkflowExecution documents in the dev DB

[aclum]

The data_generation_set records attached to https://github.com/microbiomedata/issues/issues/935 are in mongo dev and can be used for end to end testing of a workflow starting with reads qc.