v3 1st draft overview

[mr-c]

CWL descriptions for each tool

• [x] seqprep https://github.com/ProteinsWebTeam/ebi-metagenomics-cwl/pull/4
• [x] trimmomatic https://github.com/common-workflow-language/workflows/pull/132 Order of trimming steps can still be made more flexible
• [x] "biopython"

  Sequences < 100 nucleotides in length removed.

  • [x] need to find the script

• [x] hmmer
  • [x] figure out how to use https://github.com/matthiasblum/hmmer-cwl in the context of this workflow
• [x] FragGeneScan
  • [x] how is this used at EBI?
  • [x] updated at https://github.com/ProteinsWebTeam/ebi-metagenomics-cwl/pull/3
• [x] InterProScan
• [x] "QIIME"

  16s rRNA are annotated using the Greengenes reference database (default closed-reference OTU picking protocol with Greengenes 13.8 reference with reverse strand matching enabled)."

  • [x] Which parts of QIIME are used and how?

[mscheremetjew]

Hi Michael, Is this a list for your personal records or do you want us to take on any of these?

[mr-c]

This is for my personal records -- I started a contract with Rob Finn last week to do the above & other tasks. Feedback is welcome! I'm sure I'll have questions as I go.

[mscheremetjew]

I tried to identify all EMG pipeline tools (internal and external tools) when we had our hands-on meeting. My list is slightly longer though (please ignore it if Rob wants to):

• [x] Decompression of raw reads - gunzip, Linux tool
• [x] FASTA format header cleaning - inhouse Python script
• [x] SeqPrep - external tool
• [x] Co-ncatenate and decompress SeqPrep results - gunzip, Linux tool
• [x] Trimmomatic - external tool
• [x] Length filtering etc - inhouse Python script using BioPython module
• [x] Produce quality control statistics - customized external Python script provided by MG-RAST
• [x] Infernal - RNA detection - external tool (HMMER suite)
• [x] RNA masking - inhouse Python script
• [x] QIIME - wrapper script around external tools
• [x] QIIME script for OTU generation pick_closed_reference_otus.py
• [x] BIOM format tool (convert method) - external tool
• [x] BIOM format tool (summarize-table method - external tool
• [x] QIIME (Create list of observations) - Linux tools awk and sed
• [x] QIIME (Phylogenetic tree creation filter_tree.py) - QIIME package
• [x] FragGeneScan - external tool
• [x] InterProScan - inhouse Java tool
• [x] GO slim mapper - external tool Map2Slim
• [x] GenomeTools (Fast FASTA file chunking tool) - external C tool
• [x] KronaTools - wrapper script needs some investigation what the script does in detail
• [x] Summary step - needs some investigation to find out step workflow
• [x] Summary step (faselector) - inhouse developed C code
• [x] Summary step (map_fa_headers) - inhouse developed C code
• [x] Summary step (extract_sig_coords) - inhouse developed Python code
• [x] Taxonomy summary step - needs some investigation to find out step workflow
• [x] Sequence categorisation step - that one is just copying some date across
• [ ] Compression of result data - gzip, Linux tool or Python module, do no quite remember

[mr-c]

Thank you @mscheremetjew I've added checkboxes to your expanded list