[Mercury_Prep_N_Batch] Enable flu compatibility and move Mercury into its own GitHub repository

[sage-wright]

This PR closes #432.

🗑️ This dev branch should be deleted after merging to main.

:brain: Aim, Context and Functionality

The main Mercury_Prep_N_Batch task was becoming excessively long and was due to become its own Python repository, which it now is, see here!.

In addition, Mercury now processes BioSample & SRA metadata for influenza.

:hammer_and_wrench: Impacted Workflows/Tasks & Changes Being Made

This will affect the behavior of the workflow(s) even if users don’t change any workflow inputs relative to the last version : Yes

This PR is a revamp of the Mercury_Prep_N_Batch workflow, with clarifications made to input parameters, although all outputs remain the same.

:clipboard: Workflow/Task Step Changes

• Mercury processing takes place with the mercury.py command.
• A completely separate task downloads the Terra table

🔄 Data Processing

Docker/software or software versions changed:

• mercury:1.0.0 is now available and is being used.

Databases or database versions changed:

Data processing/commands changed:

File processing changed:

Compute resources changed:

➡️ Inputs

variables with name changes:

old	new
table_name	terra_table_name
workspace_name	terra_workspace_name
project_name	terra_project_name

new locations:

• vadr_alert_limit, skip_county are now both under mercury

new defaults:

• output_name is now optional and defaults to "mercury"

removed inputs:

• input_table which was only for debugging but is now longer necessary to test the task

⬅️ Outputs

none!

Curtis edit: New output: String mercury_script_version which is the version of the Mercury python script/package used within the workflow.

:test_tube: Testing

Test Dataset

NEEDS TESTING!!!

Commandline Testing with MiniWDL or Cromwell (optional)

Terra Testing

Suggested Scenarios for Reviewer to Test

Theiagen Version Release Testing (optional)

:microscope: Final Developer Checklist

• [ ] The workflow/task has been tested locally and results, including file contents, are as anticipated
• [x] The workflow/task has been tested on Terra and results, including file contents, are as anticipated
• [x] The CI/CD has been adjusted and tests are passing (to be completed by Theiagen developer)
• [x] Code changes follow the style guide

🎯 Reviewer Checklist

• [x] All impacted workflows/tasks have been tested on Terra with a different dataset than used for development
• [ ] All reviewer-suggested scenarios have been tested and any additional
• [x] All changed results have been confirmed to be accurate
• [ ] All workflows/tasks impacted by change/s have been tested using a standard validation dataset to ensure no unintended change of functionality
• [x] All code adheres to the style guide
• [x] MD5 sums have been updated
• [ ] The PR author has addressed all comments

🗂️ Associated Documentation (to be completed by Theiagen developer)

• [x] Relevant documentation on the Public Health Resources "PHB Main" has been updated
• [ ] Workflow diagrams have been updated to reflect changes

[kapsakcj]

I queued another test with Flu here: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/9976680c-dec1-4a4d-91e1-4c4df8c73aaa

Also reached out to our PHL partner to request additional feedback. Will ping back when I hear from them.

[kapsakcj]

documentation looks great, thanks for revamping it! Merging now and we will delete the dev branch in 2 weeks time