kapsakcj commented 1 year ago

~~setting as a draft for now, will provide further details soon~~

~~Still have a good bit of code to edit and needs testing in Terra before we set this as "ready to review"~~

~~EDIT: I think we need to adjust the spatyper output parsing code. Set this PR back to draft state until that is resolved.~~

I think we're almost wrapped up with code additions and testing in Terra. The only other thing would be to add the tool agravate, which we already have a WDL task for, but it would need updating and testing. I propose we delay that the addition of that tool

Added agrvate in PR #218 , ~~will mark this PR as ready for review once that is merged and the CI has been fixed~~

PR is finally ready for review

Summary

This PR adds 3 Staphylococcus aureus-specific tools to the merlin_magic subworkflow as well as TheiaProk_Illumina_SE and TheiaProk_Illumina_PE workflows.

staphopia_sccmec - "staphopia-sccmec reads Staphopia results or assembly to assign a SCCmec type."
- https://github.com/staphopia/staphopia-sccmec
spatyper - "Computational method for finding spa types."
- https://github.com/HCGB-IGTP/spaTyper
agrvate - "Rapid identification of Staphylococcus aureus agr locus type and agr operon variants"
- https://github.com/VishnuRaghuram94/AgrVATE

Changes for `staphopia_sccmec` and `spatyper`

CI checks: .github/workflows/miniwdl-check.yml and .github/workflows/pytest-workflows.yml
- Updated GH Actions workflows to more recent versions: actions/checkout@v3 and actions/upload-artifact@v3 to prevent warnings
tasks/species_typing/task_spatyper.wdl
- removed unnecessary ? from cpu input
- removed unnecessary echo for versioning
- added code to parse output TSV
- added 3 new outputs, renamed one output for clarity
- File spatyper_tsv
- String spatyper_repeats
- String spatyper_type
- String spatyper_docker
- runtime block - cpu now uses input variable
tasks/species_typing/task_staphopiasccmec.wdl
- removed optional input boolean hamming - we opted to always generate the hamming distance TSV
- lowered optional input cpu to 1 since this doesn't need many threads
- added additonal command to generate TSV with hamming distances (in addition to the normal true/false summary TSV)
- added code to parse summary TSV to produce a String output of detected matches
- added 3 new outputs, renamed one output for clarity
- renamed summary TSV File staphopiasccmec_results_tsv
- File staphopiasccmec_hamming_distance_tsv
- String staphopiasccmec_types_and_mecA_presence
- String staphopiasccmec_docker
- reduced RAM to 4GB
- cpu runtime attribute now uses input variable
tasks/taxon_id/task_gambit.wdl
- Updated to set output String merlin_tag to Staphylococcus aureus
tasks/utilities/task_broad_terra_tools.wdl
- Added all spatyper and staphoiasccmec outputs to export_taxon_tables task
workflows/wf_merlin_magic.wdl
- added call blocks for 2 new tools
- allow for user to define docker image used for staphopia_sccmec, in case there are new versions in the future
added all new outputs to both workflows/wf_theiaprok_illumina_pe.wdl and workflows/wf_theiaprok_illumina_se.wdl

Changes for `agrvate`

updated tasks/species_typing/task_agrvate.wdl
- lowered disk_size input var to 50, don't need much space
- fixed optional integer input syntax
- removed unnecessary echo
- added helpful comments throughout
- adjusted agrvate command to always use mummer with -m option since usearch is not available in the container (due to strict redistribution rules)
- added code for parsing summary TSV from agrvate
- added 6 new outputs:

    File agrvate_summary = summary TSV
    File agrvate_results = tarball of all agrvate output files
    String agrvate_agr_group = final agr group desitnation
    String agrvate_agr_match_score = confidence in agr match, higher number the better. ranges 0-15. 15 is max
    String agrvate_agr_canonical = canonical or non-canonical agrD gene
    String agrvate_agr_multiple = number of agr groups found (expect only 1 per isolate)
    String agrvate_agr_num_frameshifts = number of frameshifts found in CDS of extracted agr operon (done during mummer step)
    String agrvate_version = version
    String agrvate_docker = docker image used

lowered RAM to 4GB - again not a computationally intensive task
runtime cpu now uses input variable cpu
- added agrvate outputs to export_taxon_tables task in tasks/utilities/task_broad_terra_tools.wdl
- added agrvate to merlin_magic workflow
added optional input agrvate_docker_image for when new versions are released
- added agrvate outputs to theiaprok_Illumina_pe and SE workflows

Testing

Tested both PE and SE workflows on Terra with 4 Staphylococcus aureus Illumina samples. Both ran successfully and export_taxon_tables ran as expected as well.

kapsakcj commented 1 year ago

@cimendes FYI I think we will need to update the spatyper parsing code to account scenarios where more than one match is found (example: different hits from different contigs)

One of the samples I tested with had 2 hits in the output spatyper TSV, leading to parsing issues with the spatyper_repeats and spatyper_type output strings

Sample is SAMEA2238216 in my sandbox workspace within the staph_illumina_pe data table, also uploaded the TSV here: SAMEA2238216.tsv.txt

Let's be careful with pushing commits now, because we have 2 non-Theiagen labs that may be using this dev branch for testing. Only push commits when we're confident they will run properly in Terra, please!

kapsakcj commented 1 year ago

retested in Terra successfully after the recent changes to spatyper results parsing:

SE - https://app.terra.bio/#workspaces/cdph-terrabio-taborda-manual/curtis_sandbox/job_history/78fe80bb-929c-47ba-8ea0-70ca8635c290 PE - https://app.terra.bio/#workspaces/cdph-terrabio-taborda-manual/curtis_sandbox/job_history/6ca6c9a8-8190-4633-90aa-71b858cc9915

kapsakcj commented 1 year ago

Successful testing in Terra following the merge of PR #218 (addition of agrvate to S aureus subworkflow)

Single End: https://app.terra.bio/#workspaces/cdph-terrabio-taborda-manual/curtis_sandbox/job_history/37dde661-73fa-48b3-b439-1ddbdebc72a6

Paired end (1 failure is related to data and expected, failure not related to code changes): https://app.terra.bio/#workspaces/cdph-terrabio-taborda-manual/curtis_sandbox/job_history/c276ff93-3a08-411b-af42-d36c1b0ec2f4