pepkit / looper

A job submitter for Portable Encapsulated Projects
http://looper.databio.org
BSD 2-Clause "Simplified" License
20 stars 7 forks source link

`looper` logging cannot be captured although it goes to `stderr` #444

Closed simeoncarstens closed 8 months ago

simeoncarstens commented 8 months ago

In the HTTP API for looper we are building, we want to capture stdout and stderr and return it to the user. Unfortunately, this capturing (using contextlib.redirect_std{out,err}) seems not to be working completely with looper. As an example, in this demo branch we run the whole main() function of cli_looper.py in the above context managers, but only a very small fraction of the actual logging output is captured - see below output.

Does one of the looper / logmuse developers have any insights why that is the case and, possibly, how it could be fixed?

Also cc @zz1874

[simeon@courgette:~/projects/uva/hello_looper]$ looper run
Looper version: 1.6.0
Command: run
Using looper config (.looper.yaml).
Using default divvy config. You may specify in env var: ['DIVCFG']
Pipestat compatible: False
## [1 of 2] sample: frog_1; pipeline: count_lines
Writing script to /home/simeon/projects/uva/hello_looper/results/submission/count_lines_frog_1.sub
Job script (n=1; 0.00Gb): /home/simeon/projects/uva/hello_looper/results/submission/count_lines_frog_1.sub
Compute node: courgette
Start time: 2024-01-23 17:49:53
/home/simeon/projects/uva/hello_looper
Number of lines: 4
## [2 of 2] sample: frog_2; pipeline: count_lines
Writing script to /home/simeon/projects/uva/hello_looper/results/submission/count_lines_frog_2.sub
Job script (n=1; 0.00Gb): /home/simeon/projects/uva/hello_looper/results/submission/count_lines_frog_2.sub
Compute node: courgette
Start time: 2024-01-23 17:49:53
/home/simeon/projects/uva/hello_looper
Number of lines: 7

Looper finished
Samples valid for job generation: 2 of 2
Commands submitted: 2 of 2
Jobs submitted: 2
Captured stderr:

Captured stdout:
/home/simeon/projects/uva/hello_looper/pipeline/count_lines.sh data/frog1_data.txt
/home/simeon/projects/uva/hello_looper/pipeline/count_lines.sh data/frog2_data.txt
simeoncarstens commented 8 months ago

Turns out this is most likely the same problem as https://github.com/databio/logmuse/issues/26, so closing this in favor of that issue.