search

thasso / pyjip

JIP Pipeline System
http://pyjip.readthedocs.org
Other
19 stars 8 forks source link

named pipes/job with 2 outputs #60

Closed necrolyte2 closed 8 years ago

necrolyte2 commented 8 years ago

I'm trying to figure out how to implement the scenario where one command has 2 output streams where another command takes both of them in as input such as:

mkfifo outr1 outr2
cutadapt -o outr1 -p outr2 -q 25,25 r1.fastq r2.fastq &
bwa index ref.fasta
bwa mem ref.fasta outr1 outr2 | samtools view -bh - > out.bam
Poshi commented 8 years ago

Hello,

I'm not sure if there is a way to do it, but in any case I would strongly discourage it as a general approach because you can run into a blocking state quite easily. Scenario: first tool gives two outputs, one is a high volume output and the other is a low volume output. The second tool reads both outputs simultaneously. If the first tool fills the buffer of the first output, it will get blocked until the second tool reads from that output. But if the second tool is expecting the first tool to send something in the second output... pipeline blocked.

I see that in your case, this should not happen because both outputs should be equivalent, but there is a chance if the pipe buffers are small enough.

I know it is an ugly advice but... you are safer by writing the intermediate results on disk. Againg, in your specific case, it would be nice to see if you can get an interleaved output from cutadapt to feed only one stream to bwa.

Jordi Camps jcamps@gmail.com

-----BEGIN GEEK CODE BLOCK----- Version: 3.12 GCS/IT/S d-(+) s: a C++ UL++++() P+ L+(++) E- W++ N o? K? w++ !O M V PS++ PE- Y? PGP t 5 X R(+) tv(+) b+ DI+ D+ G e+++ h r(--) y+** ------END GEEK CODE BLOCK------

2016-01-28 20:20 GMT+01:00 Tyghe Vallard notifications@github.com:

I'm trying to figure out how to implement the scenario where one command has 2 output streams where another command takes both of them in as input such as:

mkfifo outr1 outr2 cutadapt -o outr1 -p outr2 -q 25,25 r1.fastq r2.fastq & bwa index ref.fasta bwa mem ref.fasta outr1 outr2 | samtools view -bh - > out.bam

— Reply to this email directly or view it on GitHub https://github.com/thasso/pyjip/issues/60.

necrolyte2 commented 8 years ago

I don't think that is ugly advice. I agree with you, just didn't know if there was a 'special' way to do it with jip. I also did not know that bwa accepted interleave and am now investigating that!

One last thing, pyjip is awesome! Good job