gravelcycles
commented
5 years ago Really interesting point. This would actually be a great reason to move away from subprocess.Popen(command).wait(). The main thing to understand here is buffers. Do some searching on your own to understand them and read this article: https://www.quora.com/What-is-output-buffering-in-Python
What is happening is the following (it's a bit low-level):
-
When
print("First command")is executed, it doesn't actually put the characters on the screen automatically, since doing so takes a lot of time. The letters are added to a buffer and then all written at once to the screen when enough characters are added to the buffer or something forces the buffer to be flushed. -
Next, the
subprocess.Popen(command).wait()finishes and flushes its buffer (writes the words onto the screen) when it terminates (program termination forces the buffer to be flushed). The important thing is that when a subprocess is opened, it uses a different buffer since it's in a different process. -
After that, either
print("Third command")is executed or something else, but we are back to using the original process. -
The last subprocess call doesn't write to the screen, it writes to
subprocess.PIPEand then isprinted within the original command.
WARNING: DON'T USE THIS IN THE CODEBASE UNLESS ABSOLUTELY NECESSARY. THERE ARE ALMOST ALWAYS WORK AROUNDS: Adding sys.stdout.flush() after print("First command") will force the buffer to be flushed.
Instead, I would suggest to abandoning the subprocess.Popen(command).wait() pattern and move to the subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE) pattern. Besides solving this problem, we also capture stderr output that we can now use in error messages, etc.
Note: The best I can do to explain the discrepancy between running it on your machine or on the LSF cluster is that they handle buffers differently. I'm unsure exactly what the setting is that leads to the difference. However, the above solution will work for both environments.
yunjunz
falkamelung
A problem in PYSAR is that the output from pysarApp.py gets scrambled, which makes debugging very difficult. Check out the functions below. If you run
./test_print_order.pyin interactive mode (used by Yunjun for development) you get an output as desired:But if you run as a job,
bsub < test_print_order.jobyou get the output scrambled:That makes debugging very difficult. For now we just need some advice on how to do the
subprocess.Popen submissioninpysarApp.py. I see 4 options:subprocesshas options to pipe output to the correct place?Content: of `/nethome/famelung/test_python/test_print_order.py ' :
Content: of `/nethome/famelung/test_python/test_print_order.job ' :