Workflow: flow_setup

[r06942072]

flow_setup is broken down into three parts: setup_tree download_genomic download_others

tree.cwl wget.cwl extract_md5checksums.cwl check_md5sum.cwl
gunzip.cwl

[r06942072]

Technical problem of cwl solved

• How to stack both File[] and File ?

[r06942072]

• extract from original downloaded checksums.txt

[r06942072]

• in outputs section of gunzip.cwl

• list all the possibilities of 4 datas and do not have ambiguous

• Success!!

[mpoelchau]

Genomic fasta (in_fasta) needs to be here at the end of the whole workflow: data/other_species/[gggsss]/[assembly-name]/scaffold/

Genomics gff (in_gff) needs to be here at the end of the whole workflow: data/other_species/[gggsss]/[assembly-name]/scaffold/analyses/[annotation-release-number]/

It doesn't matter (at least to me) whether the files are in those directories at the end of flow_setup or flow_apollo2 - but it would be great if they can be moved there at the end of the entire workflow of workflows.

[r06942072]

【change of code】

• In flow_setup, collecting unzip data from out_gunzip modify output section of gunzip.cwl, capturing all files only in one variable called out_gunzip. Now, not able to get rid of .gz file even if expression is applied. Link: https://github.com/NAL-i5K/CWL_Common-Workflow-Language/blob/dev/flow_setup/gunzip.cwl
  

• Treat genomic fasta(in_fasta) and genomic gff(in_gff) as external inputs. add in_fasta and in_gff in job.yml and are used in input section of flow apollo2

Difficulties:

1. the File not find. Due to output section of final-workflow.cwl is excuted in very last, so when the pipeline arrive flow_apollo, those genomics data(fasta and gff) not found.

Solution:

1. We should go back to the root problem to discuss a nicer way to capture "gff" and "fasta" from gunzip block.
   

• move genomic fasta(in_fasta) and genomic gff(in_gff) to directory tree Genomic fasta (in_fasta) needs to be here at the end of the whole workflow: data/other_species/[gggsss]/[assembly-name]/scaffold/ Genomics gff (in_gff) needs to be here at the end of the whole workflow: data/other_species/[gggsss]/[assembly-name]/scaffold/analyses/[annotation-release-number]/

Solution: append block mv_to_data in the end of flow_setup

• backup all things in flow setup

Solution: append block mv_to_data in the end of flow_setup.

• document cwl message in a log file Ref:
  1. https://askubuntu.com/questions/420981/how-do-i-save-terminal-output-to-a-file