#list addOrReplaceGroupsBam #list addOrReplaceGroupsBai does not weave in molgenis_compute/v5_20140522

[mmterpstra]

In the following example it does't weave the parameters

#list addOrReplaceGroupsBam
#list addOrReplaceGroupsBai 
for file in "${addOrReplaceGroupsBam[@]}" "${addOrReplaceGroupsBai[@]}"; do

so the result is:

#list addOrReplaceGroupsBam
#list addOrReplaceGroupsBai 
for file in  ; do

here is more documentation from previous mailing:

Bug or (unintended) feature: generation fails with multiple lists, maybe caused by inheritance of variables.

molgenis_compute/v5_20140522:

from parameters.csv (note that these files are two outputs from a single program). addOrReplaceGroupsDir,${projectDir}/addOrReplaceReadGroups/ addOrReplaceGroupsBam,${addOrReplaceGroupsDir}${internalId}${sampleName}.bam addOrReplaceGroupsBai,${addOrReplaceGroupsDir}${internalId}${sampleName}.bai

internalId / sampleName from samplesheet...

protocols/protocol.sh

#list addOrReplaceGroupsBam
#list addOrReplaceGroupsBai
...
for file in "${addOrReplaceGroupsBam[@]}" "${addOrReplaceGroupsBai[@]}"; do
    echo "getFile file='$file'"
    getFile $file
done

protocol after weave:

# Connect parameters to environment
#no variable declarations
...
#list addOrReplaceGroupsBam
#list addOrReplaceGroupsBai
...
for file in  ; do
    echo "getFile file='$file'"
    getFile $file
done

Works with single list:

molgenis_compute/v5_20140522:

#list addOrReplaceGroupsBam
...
for file in "${addOrReplaceGroupsBam[@]}"; do
    echo "getFile file='$file'"
    getFile $file
done

#######################################

after weave:

Connect parameters to environment

addOrReplaceGroupsBam[0]="path/to/bam/file.bam"
...
#list addOrReplaceGroupsBam
...
for file in "path/to/bam/file.bam"; do
    echo "getFile file='$file'"
    getFile $file
done

[mmterpstra]

also note that #list addOrReplaceGroupsBam, addOrReplaceGroupsBai works (and maybe in this context is better but this does not change the fact that this is a weave problem )

[mmterpstra]

Additional comments by RoanKanninga: Roan Kanninga roankanninga@gmail.com Mon, Jul 13, 2015 at 9:23 AM Reply-To: gcc-ngs@googlegroups.com To: George Byelas h.v.byelas@gmail.com Cc: Freerk van Dijk freerk.van.dijk@gmail.com, Niek de Klein niekdeklein@gmail.com, gcc-ngs gcc-ngs@googlegroups.com Hi guys, for me it is also not working, what is changed in compute5_20140522 compared to previous version? I know that you guys implemented this datastaging queue thing, but what else, something has changed!! It is really strange that on slurm these 2 lists are not working anymore. The list,list is ofcourse working but it is now making combinations of both the lists. I have 2 samples and 2 lists containing .bam and .bai files. With the list,list option I now get 4 files in each list, instead of 2 files in each list, strange..

Cheers, Roan [Quoted text hidden] Roan Kanninga roankanninga@gmail.com Mon, Jul 13, 2015 at 9:46 AM Reply-To: gcc-ngs@googlegroups.com To: George Byelas h.v.byelas@gmail.com Cc: Freerk van Dijk freerk.van.dijk@gmail.com, Niek de Klein niekdeklein@gmail.com, gcc-ngs gcc-ngs@googlegroups.com One small other thing, if we declare multiple lists that are not coming out of the parameters.csv file, but for example out of the samplesheet everything is fine. So it has something to do with the weaving, because the parameters out of the samplesheet are not weaved, but the parameters out of the parameters.csv do.

Roan [Quoted text hidden]

[mmterpstra]

see also: https://github.com/mmterpstra/molgenis-c5-TumorNormal/blob/devel/notes_list.txt and clone (devel tree of molgenis-c5-TumorNormal): https://github.com/mmterpstra/molgenis-c5-TumorNormal/tree/devel then look at the test/ groups dirs for the weaved examples

[mmterpstra]

The command: bash /groups/umcg-oncogenetics/tmp04/git/molgenis-c5-TumorNormal/test/molgenis-compute-core-1.0.0-SNAPSHOT/molgenis_compute.sh --generate -p /groups/umcg-oncogenetics/tmp04/git/molgenis-c5-TumorNormal/parameters.molgenis.csv -p /groups/umcg-oncogenetics/tmp04/git/molgenis-c5-TumorNormal/samplesheet.csv.tmp.csv -p /groups/umcg-oncogenetics/tmp04/git/molgenis-c5-TumorNormal/scatter_id.csv -w /groups/umcg-oncogenetics/tmp04/git/molgenis-c5-TumorNormal/workflow_test.csv --backend pbs --weave -rundir /groups/umcg-oncogenetics/tmp04/git/molgenis-c5-TumorNormal/groups/umcg-oncogenetics/tmp04/projects/something.20140402.104325-2/jobs -header /groups/umcg-oncogenetics/tmp04/git/molgenis-c5-TumorNormal/test/molgenis-compute-core-1.0.0-SNAPSHOT/templates/localhost//header.ftl -submit /groups/umcg-oncogenetics/tmp04/git/molgenis-c5-TumorNormal/test/molgenis-compute-core-1.0.0-SNAPSHOT/templates/localhost//submit.ftl -footer /groups/umcg-oncogenetics/tmp04/git/molgenis-c5-TumorNormal/test/molgenis-compute-core-1.0.0-SNAPSHOT/templates/localhost//footer.ftl

[mmterpstra]

molgenis command at: https://gist.github.com/mmterpstra/bb68291d4a4735923368

[mark-de-haan]

Not a bug, its a feature. According to the manual found here:

If users want to combine lists that coming from separated files, lists should be declared on the same line