Sanity Check: does datalad runner also buffer stdout (cannot force flush in wrapper script)

[yarikoptic]

Here is a basic (actually a "fancy" version since a generator)

```python #!/usr/bin/env python # emacs: -*- mode: python; py-indent-offset: 4; tab-width: 4; indent-tabs-mode: nil -*- # ex: set sts=4 ts=4 sw=4 noet: from datalad.runner.protocol import GeneratorMixIn from datalad.runner.utils import ( AssemblingDecoderMixIn, ) from datalad.cmd import ( GitWitlessRunner, StdOutErrCapture, ) class GeneratorStdOutErrCapture(GeneratorMixIn, AssemblingDecoderMixIn, StdOutErrCapture): """ Generator-runner protocol that captures and yields stdout and stderr. """ def __init__(self): GeneratorMixIn.__init__(self) AssemblingDecoderMixIn.__init__(self) StdOutErrCapture.__init__(self) def pipe_data_received(self, fd, data): if fd in (1, 2): print(f"processing data: {data} for {fd}") self.send_result((fd, self.decode(fd, data, self.encoding))) else: StdOutErrCapture.pipe_data_received(self, fd, data) if __name__ == '__main__': import sys, os git_runner = GitWitlessRunner() generator = git_runner.run( sys.argv[1:], protocol=GeneratorStdOutErrCapture, env = os.environ.copy(), ) for out in generator: print(f"generator output: {out}") ```

which on this simple script

#!/usr/bin/env python

from time import sleep
import sys

for i in range(5):
    sys.stdout.write(f"stdout {i}\n")
    sys.stderr.write(f"stderr {i}\n")
    sleep(0.1)

which, if stdout is not piped would just interleave

❯ ./prints.py
stdout 0
stderr 0
stdout 1
stderr 1
stdout 2
stderr 2
stdout 3
stderr 3
stdout 4
stderr 4

and if stdout redirected, wouldn't be flushed I guess so we get

❯ ./prints.py | tee /tmp/out
stderr 0
stderr 1
stderr 2
stderr 3
stderr 4
stdout 0
stdout 1
stdout 2
stdout 3
stdout 4

So datalad runner produces

❯ ./demo_datalad_capture.py ./prints.py
processing data: b'stderr 0\n' for 2
generator output: (2, 'stderr 0\n')
processing data: b'stderr 1\n' for 2
generator output: (2, 'stderr 1\n')
processing data: b'stderr 2\n' for 2
generator output: (2, 'stderr 2\n')
processing data: b'stderr 3\n' for 2
generator output: (2, 'stderr 3\n')
processing data: b'stderr 4\n' for 2
generator output: (2, 'stderr 4\n')
processing data: b'stdout 0\nstdout 1\nstdout 2\nstdout 3\nstdout 4\n' for 1
generator output: (1, 'stdout 0\nstdout 1\nstdout 2\nstdout 3\nstdout 4\n')

[asmacdo]

Thanks @yarikoptic I think this means our current approach is going to have to do. If theres anything else duct should do, please reopen.