emo-bon / MetaGOflow

MGnify oriented implementation for the Marine Genomic Observatories oriented pipeline, developed in the framework of an EOSC-Life funded project
https://metagoflow.readthedocs.io
Apache License 2.0
7 stars 7 forks source link

Chunked results should be concatenated at end of execution #10

Closed cymon closed 7 months ago

cymon commented 1 year ago

Some steps, such as InterProScan, chunk the work into bits. These should be joined after all the chunks are computed. At present the number of output files depends on the input:

If there is a lot of data it gets split up into more than one file: DBB.merged_CDS.I5_001.tsv.gz DBB.merged_CDS.I5_002.tsv.gz

When there is less output there is only one file: DBB.merged_CDS.I5.tsv.gz

cymon commented 7 months ago

Fixed in latest version