capture: combine or line-order intersparse stdout and stderr

[ssbarnea]

At this moment pytest capture is not able to combine stdout and stderr lines in a chronological order, thus the capture is separated. This means that that on larger outputs it is impossible to identify where did the stderr lines originated from (example ansible output, where warnigns are sent to stderr).

[RonnyPfannschmidt]

it might make sense to also integrate logging there in some manner,

being order aware is a critical thing in various setups for comprehending whats happening

[nicoddemus]

Hi,

How would that look like, a new cmdline option? Also how would that affect cap* fixtures and the existing sys and fd modes?

[RonnyPfannschmidt]

its not clear to me how those would interact the outerr return values turn senseless in the face of a merge to one fd

[nicoddemus]

I'm thinking merging stdout into stderr (similar to how subprocess does it) would mean that stderr would never capture anything.

Thinking more about that, perhaps cap* fixtures shouldn't be affected by this command-line at all, as plenty of tests in a test suite using those fixtures would then start failing if you used the command-line option.

[blueyed]

Just for info: I think separate threads could be used to capture stdout/stderr separately and merge them. This is what I've used with auto-stdin handling with/via ptys.

[RonnyPfannschmidt]

i don think we can get correct output order that way unless buffering and locking is strangely controlled, im -1 on trying with htreads

[blueyed]

Yes, it's not really straightforward nor easy, but would work - and there appears no better method if you do not want to merge it into a single stream really AFAIK.

[RonnyPfannschmidt]

practically speaking for pure python its reasonably sensible to just use a buffered pipe and wire up line receivers that receive lines with timestamps plus making the streams flush eagerly with a sleep(0) thrown in to trigger other threads, but will still create potential order flips

[aklajnert]

What if the lines from both streams would be merged into one deque, and kept as a tuple of message + int which would represent stdout or stderr?
It would keep the order, and separate streams would be still easily recoverable.

[bukzor]

Trying to timestamp the lines won't work -- as soon as the two pipes are separate objects in the kernel, there's no guarantees about the relative ordering of writes/reads on two separate pipes. The only way to preserve chronological order exactly (with no heuristics, edge cases, or race conditions) is to combine stderr and stdout. Luckily this is also much simpler to implement than the other ideas: os.dup2(1,2).

Is it possible for me to add a --capture=dup2 mode to py.test from conftest?

[bukzor]

This has the desired effect when used as a contextlib.contextmanager, but does nothing when used as a pytest.fixture. Is that a bug we can fix at least?

def combine_stderr():                                                                                                                                                                  
  import os                                                                                                                                                                            
  os.dup2(1, 2)                                                                                                                                                                        

  import sys                                                                                                                                                                           
  orig_stderr = sys.stderr                                                                                                                                                             
  sys.stderr = sys.stdout                                                                                                                                                              
  try:                                                                                                                                                                                 
    yield                                                                                                                                                                              
  finally:                                                                                                                                                                             
    sys.stderr = orig_stderr                                                                                                                                                           

[nicoddemus]

This has the desired effect when used as a contextlib.contextmanager, but does nothing when used as a pytest.fixture. Is that a bug we can fix at least?

I think this will work as a fixture if you disable the capture plugin entirely, as I it changes the streams before the fixture is activated.

The dup2 idea is great, but would be needed to be implemented into the capture plugin itself, I think.

[bukzor]

This has the desired effect when used as a contextlib.contextmanager, but does nothing when used as a pytest.fixture. Is that a bug we can fix at least?

I think this will work as a fixture if you disable the capture plugin entirely ...

But the desired behavior is to get chronological ordered output of the captured stdout/stderr. It's true that disabling the capture plugin entirely gives chronologically ordered output, but that was already true :) since in that case both stdout and stderr are directed to the same file descriptor, pointed at /dev/tty.

[bukzor]

The dup2 idea is great, but would be needed to be implemented into the capture plugin itself, I think.

Must that be a pytest patch, or can it be done from a user-written plugin?