Augur Updates for RSV-A and RSV-B

This PR closes #481.

🗑️ This dev branch should be deleted after merging to main.

:brain: Aim, Context and Functionality

This PR enables the Augur workflow to be able to analyze RSV-A and RSV-B assemblies. Prior to this PR, the workflow would only process SARS-CoV-2, seasonal Influenza A and B, and Mpox.

The Augur Prep workflow has not been updated as there was no need for changes therein.

:hammer_and_wrench: Impacted Workflows/Tasks & Changes Being Made

This will affect the behavior of the workflow(s) even if users don’t change any workflow inputs relative to the last version : Yes

Previously, for seasonal Influenza and Mpox, augur reference files and default values were set in the augur_utilities task. With the move to house all organism specific defaults to the organism_parameters workflow, we have now moved the default augur parameters for seasonal Influenza and Mpox to the organism_parameters workflow.

SARS-CoV-2 augur defaults have not been moved yet as the current behaviour is to download the augur defaults directly from nextstrain's "https://github.com/nextstrain/ncov" page. As SARS-CoV-2 clades tsv file is updated quite frequently with new lineages, this is the best strategy. In the future, the other pathogens may also take this direction.

Running this workflow on different occasions could result in different results, e.g. due to use of a live database, "latest" docker image, or stochastic data processing : Yes, if the augur reference files are updated in the future.

:clipboard: Workflow/Task Step Changes

🔄 Data Processing

Docker/software or software versions changed: No

Databases or database versions changed: No

Data processing/commands changed: Yes, organism default sources moved.

File processing changed: Yes, organism default sources moved.

Compute resources changed: No

➡️ Inputs

Exposed the following optional inputs, used in augur_frequencies, but not currently used in the workflow:

augur   min_date    Float
augur   pivot_interval  Int
augur   proportion_wide Float
augur   narrow_bandwidth    Float

The following optional inputs have been removed:

flu_defaults    cpu Int
flu_defaults    disk_size   Int
flu_defaults    docker  String
flu_defaults    flu_lat_longs_tsv   File
flu_defaults    memory  Int
mpxv_defaults   cpu Int
mpxv_defaults   disk_size   Int
mpxv_defaults   docker  String
mpxv_defaults   memory  Int
mpxv_defaults   mpxv_auspice_config File
mpxv_defaults   mpxv_clades_tsv File
mpxv_defaults   mpxv_lat_longs_tsv  File
mpxv_defaults   mpxv_reference_fasta    File
mpxv_defaults   mpxv_reference_genbank  File

The following optional inputs are exposed in the workflow but not used for this workflow:

organism_parameters gene_locations_bed_file File
organism_parameters genome_length_input Int
organism_parameters hiv_primer_version  String
organism_parameters kraken_target_organism_input    String
organism_parameters nextclade_dataset_name_input    String
organism_parameters nextclade_dataset_tag_input String
organism_parameters pangolin_docker_image   String
organism_parameters primer_bed_file File
organism_parameters reference_gff_file  File
organism_parameters vadr_max_length Int
organism_parameters vadr_mem    Int
organism_parameters vadr_options    String
organism_parameters vadr_skip_length    Int

⬅️ Outputs

None changed

:test_tube: Testing

Test Dataset

Commandline Testing with MiniWDL or Cromwell (optional)

Locally tested using the RSV-A dataset, see Terra testing below, and works as expected.

Terra Testing

As we moved the augur pathogen specific parameters to the organism parameters workflow, we tested all previous and the new pathogens to ensure that those still work.

Suggested Scenarios for Reviewer to Test

It was noticed that the augur_refine step fails when the date information does not align with the genetic divergence. For example, the RSV-A sequences were initially given dates in 2023 and would fail this step. However, when the dates were corrected to the appropriate collection dates for the samples in 2013-2014, the workflow was able to run to completion successfully. successful run: https://app.terra.bio/#workspaces/theiagen-training-workspaces/Theiagen_Otieno_Sandbox/job_history/fa391728-0fa6-448f-9c4a-64cddc60c9ae failed run: https://app.terra.bio/#workspaces/theiagen-training-workspaces/Theiagen_Otieno_Sandbox/job_history/0b37dff6-5ee5-4bb5-ab11-285018de7ba0

The reviewer can test any additional appropriately dated sequences to see if there are any failures.

Theiagen Version Release Testing (optional)

:microscope: Final Developer Checklist

[x] The workflow/task has been tested locally and results, including file contents, are as anticipated
[x] The workflow/task has been tested on Terra and results, including file contents, are as anticipated
[x] The CI/CD has been adjusted and tests are passing (to be completed by Theiagen developer)
[x] Code changes follow the style guide

🎯 Reviewer Checklist

[x] All impacted workflows/tasks have been tested on Terra with a different dataset than used for development
[x] All reviewer-suggested scenarios have been tested and any additional
[x] All changed results have been confirmed to be accurate
[x] All workflows/tasks impacted by change/s have been tested using a standard validation dataset to ensure no unintended change of functionality
[x] All code adheres to the style guide
[x] MD5 sums have been updated
[x] The PR author has addressed all comments

🗂️ Associated Documentation (to be completed by Theiagen developer)

[x] Relevant documentation on the Public Health Resources "PHB Main" has been updated
[x] Workflow diagrams have been updated to reflect changes

Apologies for the delay on reviewing, this took quite some time to test across the different organisms and troubleshoot when user error (👨 ➡️ 🪞) occurred. Thank you for your patience.

I have no further code changes to suggest or request ✅

Thank you for updating the documentation in Notion 👍

Here are my test workflows and a short description of observed behavior:

sars-cov-2:
- augur_prep ✅ : https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/c7aded86-d835-49bc-88c0-597373648d7d
- Augur ✅ : https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/c7aded86-d835-49bc-88c0-597373648d7d
- auspice JSON viewable without issues ✅ Can color tree nodes by both nextclade clade and pango lineage
Mpox:
- augur_prep ✅ https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/fde787cb-eda8-4b8b-9300-536396a7ff68
- augur ✅ https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/242fc2c6-dfad-4002-b565-62332bdb6e7e
- notes: auspice_input_json shows clade_membership is unassigned for all samples. But we can overlay the merged-metadata.tsv file to color by clades in auspice
Flu:
- augur_prep ✅: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/49fa61f1-c4e3-43b0-a488-2ad7d19187ab
- augur ✅: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/1950f4c9-1745-4c79-b8b5-693471dec9b0
- notes: H3N2 samples. Was able to visualize the auspice_input_json and color clades after overlaying merged-metadata.tsv file. 💡 Reminder to users: The user should set organism = flu and set the flu_segment to HA or NA and additionally set flu_subtype to H1N1 or H3N2. This was not immediately clear to me and I spent time troubleshooting.
rsv-a:
- augur_prep ✅: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/61b94c5b-76dc-4af5-81de-e7ebde9d0288
- augur ✅: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/ab12c336-1e93-4fb0-800c-eb18cb42a6a5
- notes: auspice_input_json shows clade_membership for 1 sample included. 2 samples pruned during wf (I'm not terribly surprised based on my poor choice of test samples). But can always overlay clade membership via meatdata TSV
rsv-b:
- augur_prep ✅: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/d943bb81-3165-4b24-8f2e-5f58da352581
- augur ✅: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/0df3e391-4530-47b0-a658-6ce7907781fb
- notes: Ran successfully, but only after setting distance_tree_only to true. I think it failed previously on augur_refine step because of the very odd makeup of the samples (and their respective qualities) I included in my set. What matters is that the workflow ran, and used the reference materials that are now set by default in the organism parameters subworkflow.

theiagen / public_health_bioinformatics