search

nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.74k stars 626 forks source link

Explore a replacement for double tee processes #3859

Closed pditommaso closed 1 year ago

pditommaso commented 1 year ago

When nextflow launch a job execution runs two tee background processes to split the command output into stdout and stderr streams.

https://github.com/nextflow-io/nextflow/blob/8338fe608e9e60b104c273cb44b95f5ec9f1fb86/modules/nextflow/src/main/resources/nextflow/executor/command-run.txt#L150-L153

This however can cause the cause the execution to hang in some conditions hard to replicate. See for example https://github.com/nextflow-io/nextflow/issues/1760.

The goal of this task it replace the use of tee with an alternative approach.

pditommaso commented 1 year ago

Nice trick here

(cmd | tee stdout.log) 3>&1 1>&2 2>&3 | tee stderr.log

the problem is how to capture the error status when there's an error

pditommaso commented 1 year ago

Bingo !

(set -o pipefail; (pizza-with-ananas | tee stdout.log) 3>&1 1>&2 2>&3 | tee stderr.log)
pditommaso commented 1 year ago

Interesting the resulting process tree 

-+= 22627 pditommaso -bash
 |-+= 79034 pditommaso /Users/pditommaso/.sdkman/candidates/java/current/bin/java -Dfile.encoding=UTF-8 -XX:+TieredCompilation -XX:TieredStopAtLevel=1 --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio.file.spi=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.fs=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.ftp=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.file=ALL-UNNAMED --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED -cp :/Users/pditommaso/Projects/nextflow/modules/nf-httpfs/build/libs/nf-httpfs-23.04.0.jar:/Users/pditommaso/Projects/nextflow/modules/nf-commons/build/libs/nf-commons-23.04.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy-nio/3.0.16/796fa50e3465f62df1d5f2db0b32d279f03c9d60/groovy-nio-3.0.16.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy-templates/3.0.16/6c8a70a8e287ff142f076c8f212ee612b7ce4c93/groovy-templates-3.0.16.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy-xml/3.0.16/b4735fa9002e2d820f3d83e24e814b7067c932cb/groovy-xml-3.0.16.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy-json/3.0.16/bed828781dc84fd9d4fe96fb6bfbd8e0725cdf23/groovy-json-3.0.16.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.codehaus.groovy/groovy/3.0.16/5708631024a25c56487dc1211d239cc916001455/groovy-3.0.16.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.slf4j/jcl-over-slf4j/1.7.36/d877e195a05aca4a2f1ad2ff14bfec1393af4b5e/jcl-over-slf4j-1.7.36.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.slf4j/jul-to-slf4j/1.7.36/ed46d81cef9c412a88caef405b58f93a678ff2ca/jul-to-slf4j-1.7.36.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.slf4j/log4j-over-slf4j/1.7.36/2a753acda077203a4794f106871bb237501c9a53/log4j-over-slf4j-1.7.36.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/ch.qos.logback/logback-classic/1.2.11/4741689214e9d1e8408b206506cbe76d1c6a7d60/logback-classic-1.2.11.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/ch.qos.logback/logback-core/1.2.11/a01230df5ca5c34540cdaa3ad5efb012f1f1f792/logback-core-1.2.11.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.codehaus.gpars/gpars/1.2.1/c3ea0fbcd67a163bd5e3a3efdaa3428262d0d437/gpars-1.2.1.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/ch.artecat.grengine/grengine/3.0.0/a882ab181d64a08d4c65e705639b10bda40e91c3/grengine-3.0.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/commons-lang/commons-lang/2.6/ce1edb914c94ebc388f086c6827e8bdeec71ac2/commons-lang-2.6.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.pf4j/pf4j-update/2.3.0/27b5ac44b6017013ed29727383026b592c351478/pf4j-update-2.3.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.15/49d94806b6e3dc933dacbd8acb0fdbab8ebd1e5d/commons-codec-1.15.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.11.0/a2503f302b11ebde7ebc3df41daebe0e4eea3689/commons-io-2.11.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.beust/jcommander/1.35/47592e181b0bdbbeb63029e08c5e74f6803c4edd/jcommander-1.35.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.esotericsoftware.kryo/kryo/2.24.0/c6b206e80cfd97e66a1364003724491c757b92f/kryo-2.24.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.iq80.leveldb/leveldb/0.12/12ea5d0e5640d91695210bfb065562ee969a25ff/leveldb-0.12.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.eclipse.jgit/org.eclipse.jgit/6.2.0.202206071550-r/d59e7ae8528fcc3854acbb307de8443b28a5c945/org.eclipse.jgit-6.2.0.202206071550-r.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/javax.mail/mail/1.4.7/9add058589d5d85adeb625859bf2c5eeaaedf12d/mail-1.4.7.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/javax.activation/activation/1.1.1/485de3a253e23f645037828c07f1d7f1af40763a/activation-1.1.1.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.33/2cd0a87ff7df953f810c344bdf2fe3340b954c69/snakeyaml-1.33.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.jsoup/jsoup/1.15.3/f6e1d8a8819f854b681c8eaa57fd59a42329e10c/jsoup-1.15.3.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/jline/jline/2.9/84693d4d75aa2a3af8447873cac8c0aed25ce863/jline-2.9.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.pf4j/pf4j/3.4.1/6ba44242fa925f48af1784b3e57d9ab7fd186215/pf4j-3.4.1.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/dev.failsafe/failsafe/3.1.0/3a5bda837103d319e841f8fe7a3eb98f2bdb480a/failsafe-3.1.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/31.1-jre/60458f877d055d0c9114d9e1a2efb737b4bc282c/guava-31.1-jre.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.36/6c62681a2f655b49963a5983b8b0950a6120ae14/slf4j-api-1.7.36.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.multiverse/multiverse-core/0.7.0/db77d55199bc5672f05f5d725b70dd10033251ed/multiverse-core-0.7.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.codehaus.jsr166-mirror/jsr166y/1.7.0/8547fcb1c29b4f8c745c3f49a536aca58fc30f54/jsr166y-1.7.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.objenesis/objenesis/2.1/87c0ea803b69252868d09308b4618f766f135a96/objenesis-2.1.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.iq80.leveldb/leveldb-api/0.12/c97c934e9de3be7b48f6677385e1294c9ec25cc6/leveldb-api-0.12.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.googlecode.javaewah/JavaEWAH/1.1.13/32cd724a42dc73f99ca08453d11a4bb83e0034c7/JavaEWAH-1.1.13.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.github.zafarkhaja/java-semver/0.9.0/59a83ca73c72a5e25b3f0b1bb305230a11000329/java-semver-0.9.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.google.guava/failureaccess/1.0.1/1dcf1de382a0bf95a3d8b0849546c88bac1292c9/failureaccess-1.0.1.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.google.guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/b421526c5f297295adef1c886e5246c39d4ac629/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/3.0.2/25ea2e8b0c338a877313bd4672d3fe056ea78f0d/jsr305-3.0.2.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.checkerframework/checker-qual/3.12.0/d5692f0526415fcc6de94bb5bfbd3afd9dd3b3e5/checker-qual-3.12.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.11.0/c5a0ace696d3f8b1c1d8cc036d8c03cc0cbe6b69/error_prone_annotations-2.11.0.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.3/ba035118bc8bac37d7eff77700720999acd9986d/j2objc-annotations-1.3.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/com.google.code.gson/gson/2.2.4/a60a5e993c98c864010053cb901b7eab25306568/gson-2.2.4.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/Users/pditommaso/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.10/4b95f4897fa13f2cd904aee711aeafc0c5295cd8/commons-codec-1.10.jar:/Users/pditommaso/Projects/nextflow/modules/nextflow/build/libs/nextflow-23.04.0.jar nextflow.cli.Launcher run t.nf
   \-+- 79049 pditommaso /bin/bash -ue .command.run
     \-+- 79056 pditommaso /bin/bash -ue .command.run
       |-+- 79057 pditommaso /bin/bash -ue .command.run
       | |-+- 79059 pditommaso /bin/bash -ue .command.run
       | | \-+- 79061 pditommaso /bin/bash -ue /Users/pditommaso/Projects/nextflow/work/ee/9d64e9f981f9718000ab2a0883dfe3/.command.sh
       | |   \--- 79063 pditommaso sleep 300
       | \--- 79060 pditommaso tee .command.out
       \--- 79058 pditommaso tee .command.err