Simplify the pipeline

[laughedelic]

Everybody knows that launching loquats is not so fast and managing the pipeline manually is not convenient at all. So I suggest some simplification to the pipeline.

• blast results merging is a bottleneck of the pipeline, because it cannot me parallelised more: one task/worker per sample
• counting loquat is very light-weight and there is no point in launching it in a separate step
• assignment loquat already needs a lot of memory

So I suggest to merge these three steps in one loquat:

1. merge blast results
2. do assignment
3. do counting

Again, this works as task per sample.

Same reasons to merge flash and split step. But this requires having split as a separate step for pipelines that don't need paired-end reads merging.

[marina-manrique]

Fine for me about doing in the same task: merge, count, assign

Same reasons to merge flash and split step. But this requires having split as a separate step for pipelines that don't need paired-end reads merging.

and this I would also prefer to have it in independent steps so we can work with single reads/scaffolds...

[laughedelic]

and this I would also prefer to have it in independent steps so we can work with single reads/scaffolds...

what I mean is having different pipelines for these cases:

For paired-end reads:

1. flash + split-on-chunks
2. blast
3. merge-chunks + assign + count

And for non paired-end reads/scaffolds:

1. split-on-chunks
2. blast
3. merge-chunks + assign + count

(the difference is only in the first loquat/step).

@marina-manrique @eparejatobes I'd like to get feedback from you, because if you support this suggestion, I want to include it in the next release:

• it's easy to implement
• it makes usage of MG7 much simpler (3 steps instead of 5/6)
• it fastens the process, because
  • split step is very stupid and fast, no point in waiting for a separate loquat launch to do it
  • same about merge
  • same about counting, which is super fast (like 10 seconds in the meta-gacu with scaffolds) and it shares the same environment with the assignment step

[marina-manrique]

Perfect for me

[eparejatobes]

OK, :+1: