Alpha implementation of end-to-end workflow

[georgiesamaha]

Not ready for a merge. This PR covers alpha implementation of end-to-end workflow from porechop/nanoplot though to multiqc report creation

[georgiesamaha]

Issues in development scripts have prevented me from going further. In order to implement initial QC, contamination screen, and assemblies the following things need to be clarified:

• No explanation on how to set up input directory which capture pycoqc and nanoplot inputs
• Multiple scripts for nanoPlot but neither are implemented in the run script. Which one should be run? This one or this one?
• PycoQC isn't implemented in the run script.
• Kraken2 contamination screen script has lots of hashed out code, I'm unsure what needs to be implemented, what has yet to be tested, and reasoning behind this structure. Additionally, some parts require manual intervention without explanation.

[nandan75]

I have made changes and pushed them to this branch

1. No explanation on how to set up input directory which capture pycoqc and nanoplot inputs
2. PycoQC isn't implemented in the run script.

• These two steps are to be run once per "experimental run/batch i.e. all samples

• Input to both the module is *raw_data/sequencing_summary.txt** which is a file generated by the ONT sequencing run itself.

• When testing the bash pipeline, I have kept the above two scripts out of the once-per sample run_pipeline.sh. The input sequencing_summary*.txt

  • Can be placed in the same folder raw_data as the rest of the individual (per sample barcode) .zip files.

• The run_pipeline.sh script is run once-per-sampleID. So there is no mention of the above two modules.

3. Multiple scripts for nanoPlot but neither are implemented in the run script. Which one should be run? This one or this one?

   • Please use the script run_nanoplot.nf for nanoPlot from the nextflow repo which represents this script
   • Please use the script run_porechop.nf for porechop from the nextflow repo which represents this script.

4. Kraken2 contamination screen script has lots of hashed out code, I'm unsure what needs to be implemented, what has yet to be tested, and reasoning behind this structure. Additionally, some parts require manual intervention without explanation.

   • I have now cleaned the script of the unwanted #ed lines. The script for kraken2 in this repo is here

[georgiesamaha]

Currently stuck at select_assembly step. Issue with parsing input trycycler cluster files to the python script without overwriting other clusters. Have started on docs/ for specific instructions re: execution on Gadi while we're still testing.

[georgiesamaha]

Have hit an issue with medaka_polish_flye process.

Error description Process medaka_polish_flye fails to index assembly fasta due to empty Chr_contigs/flyeChromosomes.fasta file for barcode10. This sample is expected to fail consensus and proceed to flye only assembly. Might there be a problem with select_assembly script not outputting the right files?

Relevant files

# workdir
/scratch/er01/gs5517/workflowDev/ONT-bacpac-nf/work/98/1bf0a1f57d8492b404145cd9c53f1c/barcode10_flye_assembly/Chr_contigs/flyeChromosomes.fasta

# select_assembly.py
/scratch/er01/gs5517/workflowDev/ONT-bacpac-nf/bin/select_assembly.py

# test run script
/scratch/er01/gs5517/workflowDev/ONT-bacpac-nf/test/run_test.sh

[nandan75]

Hi @georgiesamaha

I think I have debugged the issue. Please test the pipeline now

• The issue was about breaking from the function get_all_chromosomal_contigs_using_genome_size without adding the required chromosomal contig.
• I have debugged the step independant of the actual nextflow run. Please re-run this step to confirm.
• Barcode10 is marked as "Consensus" and hence it should go though rest of the trycycler steps.

[nandan75]

Hi @georgiesamaha

• I have pushed all changes (not just the Flye-only related ones) to the branch
• Since I changed the output from select_assembly.nf process, the inputs to all subsequent channels is different.
• If you don't have to, please do not pull and test these changes.
• They should work, but then, as discussed I will try and work on the code to do it the right way (going back to the _final directory etc, once I feel better by tomorrow.

Thanks

[nandan75]

• Multiple reconciled-CHR clusters belonging to a sampleid were handled (previously seen to be missing at the msa step and thereafter).

• Pipeline successfully tested for multiple samples from raw reads to busco for

  • Consensus/Flye-only
  • Species types

• Steps still to be integrated (will work this week to finish)

  • Plasmids - plassembler
  • AMR/virumence
  • Phylogeny+heatmap plot
  • multiqc

• Encountered a problem with barcode13 (possibly memory related) - to be debugged.

• Code needs refinement and documentation - will work in coming week.

[nandan75]

Added and tested processes

• quast (today)

• bakta

• busco

• amrfinderplus (today)

• publishDir used for a few results for testing purposes - Can be shuffled later as required

• Pipeline tested for raw reads to amrfinderplus for barcode01/03/05/06/10

Next: To work on multiqc pipeline

[nandan75]

• Worked on Phylogeny tree creation using orthofinder
  • Renamed Protein files by both sampleID and species
  • Moved Protein files to single inout folder (as required by orthofinder)
  • Tested orthofinder , sccessful
  • Reference strain protein files (to be included for orthofinder) - pending - Some file name issue observed in the get_ncbi script - Will sort it tomorrow

[nandan75]

• Phylogeny tree generation (newick tree) using orthofinder works well with

  • Experiment samples included in the run.
  • Primary reference strains from each bacterial species identified for the experiment samples

• TDB

  • Phylogeny tree visualisation
  • abricate script
  • Heatmap for AMR/virulence genes from amrfidnerplus/abricate (vfdb) o/ps
  • Merge the above two to generate a composite heatmap
  • multiQC which also integrates the above image

• Plassembler script to be integrated

• Debug issue with barcode 13!

[nandan75]

• Worked on
  • amrfinder gene matrix (consenus, flye-only, reference strains) DONE
  • abricate-vfdb gene matrix (consenus, flye-only, reference strains) DONE
  • Phylogeny-heatmap image generated
    • Fontconfig error: No writable cache directories null device 1
  • No text displayed
  • Image created perfectly when script run in bash/R
  • Some additional nextflow related cache issue (as first guess from google searches)

[nandan75]

• Added/tested multiqc_config as input parameter (as of now in test/run_test.sh
• Tested multiqc
  • Works for multiqc-supported modules such as
    • kraken2
    • quast
    • bakta
    • busco

To be worked on

• pycoqc , nanoplot (and then add them to multiqc)
• plassembler (and then add them to multiqc)
• Identify issue and resolve with the combined image of phylogeny-heatmap
• Issue with barcode13 (possibly insufficient memory related) - but no time to debug

[nandan75]

• pycoqc , nanoplot tested - work out of nanoplot added to multiqc
  • Selected plots from pycoqc output to be added - In progress
• Plassembler works : The script by GS ran as is.
• There was a conflict with a older version of plassembler singularity image. This was repalced with the required image
• Bakta and busco work with plassembler output.
• Outputs passed on to multiqc and they work
  • Not too sure if busco outputs add value (or rather confuse) in the multiqc plot.
  • We can see the results and make changes.

[nandan75]

A few things pending as of 18/07/24 (1) Phylogeny-heatmap image generated (to be added to multiqc) - but an issue

• Fontconfig error: No writable cache directories null device 1

(2) Selected plots from pycoqc output to be added to multiqc - Integration of script in progress

(3) barcode13 related (possible) memory error to be investigated and resolved

Anything else!!?

[nandan75]

Hi @georgiesamaha

Good morning.

• Please test a run with all defaults bash test/run_test.sh or qsub test/run_test.sh when you feel free and let me know if it breaks.
• A couple of additional input files are included in the bash test/run_test.sh file itself (for now). We can update the pipeline to ask the sequence_summary* as an input parameter.
• The pipeline executes from reads-to-multiqc.html
• The font-display issue with phylogeny-heatmap.png image is still unresolved. I will try and look at this morning. The image without fonts is included in the multiqc report.
• Please use all/subset of samples as in the in path in bash test/run_test.sh.
• Some multiqc.html text formatting is still pending.

regards

[georgiesamaha]

All processes functional 🎉

Next step: organising results and docs.