Closed ewels closed 9 years ago
I think the core will need a more substantial update than just changing --split_files
as we don't know how many files there will be to merge and so on.
Maybe instead of looping through the list of files with
for (my $i = 0; $i <= $#files; $i++){
# Add files to run file
my $max_i = $i + $SPLIT_FILES;
my $counter = 0;
for (; $i < $max_i; $i++){
...
}
}
We should instead make a hash of arrays of files, based on both @merge_regex
and then $SPLIT_FILES
(which will count the merged files rather than the input files).
Core code written, now testing.
Tested and happy.
Need to test when regex doesn't match anything. Or when no regex is present.
Need to improve gzip concatenation due to slide from @s-andrews:
cat seq1.fq.gz seq2.fq.gz > all1.fq.gz
Some decompressors (gzip for example) will read all of the data from all1.fq.gz, but others (the java GZipInputStream class for example) will not and will silently finish at the the end of the first concatenated file.
Done a bunch of testing and bugfixing, happy that this is fairly stable now.
Would be cool if Cluster Flow could automatically recognise fastq files that should be merged before processing.
@merge_regex
to config so that people can define their own filename matches--split_files