search

ELITR / pipeliner

A tool to connect components together
0 stars 0 forks source link

stdbuf -oL tee #3

Open Gldkslfmsd opened 3 years ago

Gldkslfmsd commented 3 years ago

Hello,

I noticed that tee is missing stdbuf in the output. Then, the pipeline stucks with no error after some time. Sometimes shortly after beginning, sometimes later.

I'm attaching 2 pipeline.sh scripts: FIXED.sh.txt -- this one is fixed. I edited it to insert stdbufs. It never stucks.

This is a wrong, current piplener output. Stucks after some time. WRONG.pipeline.sh.txt

Also, there there has to be stdbuf -o0 if the commands input and output is binary (it means 0 bytes buffer), and stdbuf -oL if it is text (1 full line buffer). Maybe it would work with -o0 anytime, but I'm not sure.

Gldkslfmsd commented 3 years ago

./auto_switcher.py is an empty py script for now, I'm going to implement it.

Gldkslfmsd commented 3 years ago

my.py.txt This script is generating it.

srdecny commented 3 years ago

@Gldkslfmsd Hi, sorry for the late response. I have pushed a fix to the main branch, can you please confirm it works for you now?

Gldkslfmsd commented 3 years ago

Hi, it's better, but there are still some tees without stdbuf:

stdbuf -oL nc -lk 5001 2> >(ts '[%Y-%m-%d %H:%M:%S]' > /pwd/logs/$DATE/00-cs_audio_rec.err) | tee >(while ! ss -ltn |
Gldkslfmsd commented 3 years ago

I can't re-run the pipeline, it takes too much time to pull, restart the workers etc. I only looked at the code. You can check cruise-control/pipeliner-setups/202109-dominik whether it runs for couple of minutes without stuck. If yes, then stdbuf might be the problem. (Don't forget to remove the committed cruise-control/pipeliner-setups/202109-dominik/pipeliner.py with my simple temporary fix.)

srdecny commented 3 years ago

Pushed another commit