[TheiaCoV_Illumina_PE & _ONT] Create sub-workflow for flu-specific modules

[sage-wright]

This PR closes #409.

*** Merge after PR #468

🗑️ This dev branch should be deleted after merging to main.

:brain: Aim, Context and Functionality

Flu is a different beast due to its segmentation. The amount of flu-specific modules were growing to consume most of the workflow files for TheiaCoV_Illumina_PE and _ONT. In addition, the Nextclade outputs for HA did not have the _HA suffix, which led to confusion.

:hammer_and_wrench: Impacted Workflows/Tasks & Changes Being Made

This will affect the behavior of the workflow(s) even if users don’t change any workflow inputs relative to the last version : Yes but no

Outputs will remain the same, but the processing of the flu-specific tasks will occur in a different subworkflow

Running this workflow on different occasions could result in different results, e.g. due to use of a live database, "latest" docker image, or stochastic data processing : No

The following tasks were moved to the flu_track subworkflow:

• IRMA
• assembly statistics for the HA & NA segments
• abricate_flu
• Nextclade for both the NA & HA segment
  • the nextclade output parser task for each segment as well
• the flu_antiviral_substitutions subworkflow

:clipboard: Workflow/Task Step Changes

🔄 Data Processing

Docker/software or software versions changed:

• No changes

Databases or database versions changed:

• No changes

Data processing/commands changed:

• Consolidates multiple modules into a single subworkflow
  • all task options are still available for user modification

File processing changed:

• No changes

Compute resources changed:

• No changes

➡️ Inputs

• Flu-specific input parameters appear under the flu_track task instead of the respective task name

⬅️ Outputs

New outputs:

• File? nextclade_json_flu_ha
• File? auspice_json_flu_ha
• File? nextclade_tsv_flu_ha
• String? nextclade_ds_tag_flu_ha
• String? nextclade_aa_subs_flu_ha
• String? nextclade_aa_dels_flu_ha
• String? nextclade_clade_flu_ha
• File? nextclade_json_flu_na
• File? auspice_json_flu_na
• File? nextclade_tsv_flu_na
• String? nextclade_ds_tag_flu_na
• String? nextclade_aa_subs_flu_na
• String? nextclade_aa_dels_flu_na
• String? nextclade_clade_flu_na

Changed outputs:

• All outputs from Nextclade without a flu* suffix are NON-FLU outputs
  • Nextclade_version and Nextclade_docker are the exceptions

:test_tube: Testing

Test Dataset

Commandline Testing with MiniWDL or Cromwell (optional)

Terra Testing

ONT test here with various organisms] Illumina PE test here with various organisms

Suggested Scenarios for Reviewer to Test

• flu
• non-flu

Theiagen Version Release Testing (optional)

:microscope: Final Developer Checklist

• [x] The workflow/task has been tested locally and results, including file contents, are as anticipated
• [x] The workflow/task has been tested on Terra and results, including file contents, are as anticipated
• [x] The CI/CD has been adjusted and tests are passing (to be completed by Theiagen developer)
• [x] Code changes follow the style guide

🎯 Reviewer Checklist

• [x] All impacted workflows/tasks have been tested on Terra with a different dataset than used for development
• [x] All reviewer-suggested scenarios have been tested and any additional
• [x] All changed results have been confirmed to be accurate
• [x] All workflows/tasks impacted by change/s have been tested using a standard validation dataset to ensure no unintended change of functionality
• [x] All code adheres to the style guide
• [x] MD5 sums have been updated
• [x] The PR author has addressed all comments

🗂️ Associated Documentation (to be completed by Theiagen developer)

• [x] Relevant documentation on the Public Health Resources "PHB Main" has been updated
• [x] Workflow diagrams have been updated to reflect changes

[cimendes]

Code changes are solid 🏅 Testing after #468 is merged

[kapsakcj]

My tests here, still need to review outputs:

• ILMN PE: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/0e8c3f81-c338-488f-b62a-f3eb52a66c56
  • ✅ outputs look good, I really like the separation & clear designation of nextclade outputs
• ONT: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/3a52b0ca-ce03-47ca-a72d-b9534bece164
  • ✅ outputs look good here.
  • ⚠️ slight warning - I did have a hiccup with 2 H1N1 ONT datasets, where the IRMA assemblies for HA and NA were too different from the nextclade reference genome. I had 1 H1N1 sample run successfully and generate outputs as expected, but for the other 2 samples nextclade produced this in the log file (see below) & the nextclade string outputs for things like nextclade_aa_subs_ha and nextclade_aa_dels_ha were just NA.
  • workflow still ran successfully but just FYI in case users run into this

log file:

nextclade_ordered_writer.rs:171: In sequence #0 'SRR13439799_A_HA_H1': When processing sequence #0 'SRR13439799_A_HA_H1': When calculating seed matches: Unable to align: seed alignment covers 9.73% of the query sequence, which is less than expected 10.00% (configurable using 'min seed cover' CLI flag or dataset property). This is likely due to low quality of the provided sequence, or due to using incorrect reference sequence.. Note that this sequence will not be included in the results.

non-flu:

• Mpox ILMN PE: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/ae7f33bb-1aa2-4f13-bdf6-94a5a1a308ff
  • ✅ outputs look good, no oddities noticed
• sars-cov-2 ILMN PE: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/64359e84-8c7a-4f9f-a33a-be35e03cfca3
  • ✅ outputs look good, no oddities noticed

[kapsakcj]

Curtis TODOs:

• [x] update workflow diagram - no longer just tamiflu, state "antiviral" instead of "tamiflu" in the orange block on left side of theiacov diagram
• [x] update PHB Main docs
• [x] test with at least 1 sars-cov-2 sample
• [x] test with Mpox