Open tianon opened 7 years ago
Ok, here's an additional data point that's interesting; I can reproduce down to Docker 1.0 :sob:
$ ./output-buffering.sh
Docker 17.03: unexpected result: 2
4
1
3
Docker 1.13: unexpected result: 1
3
2
4
Docker 1.12: unexpected result: 2
1
3
4
Docker 1.11: unexpected result: 1
3
2
4
Docker 1.10: unexpected result: 2
4
1
3
Docker 1.9: unexpected result: 1
3
2
4
Docker 1.8: unexpected result: 1
3
2
4
Docker 1.7: unexpected result: 1
3
2
4
Docker 1.6: unexpected result: 1
2
4
3
Docker 1.5: unexpected result: 1
3
2
4
Docker 1.4: unexpected result: 1
3
2
4
Docker 1.3: unexpected result: 1
23
4
Docker 1.2: unexpected result: 1
2
4
3
Docker 1.1: unexpected result: 2
4
1
3
Docker 1.0: unexpected result: 1
3
2
4
(That was using essentially host-docker save busybox:latest | dind-docker load
, followed by capturing the output of dind-docker run --rm busybox awk 'BEGIN { print "1" > "/dev/stdout"; fflush("/dev/stdout"); print "2" > "/dev/stderr"; fflush("/dev/stderr"); print "3" > "/dev/stdout"; fflush("/dev/stdout"); print "4" > "/dev/stderr"; fflush("/dev/stderr"); exit }' 2>&1
)
ping @dperny this is what I was talking about last week :smile: (thanks @tianon, good timing, haha)
Was on PTO last week, added this to my queue to review.
I've got this error frequently when I use python interpreter on docker, which >>>
is printed as STDERR.
[pjknkda@anne]~% docker run -i --rm python:3.6 python -i
Python 3.6.0 (default, Feb 28 2017, 22:20:46)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = 'asdf'
>>> a
'asdf'
>>> a
>>> 'asdf'
If I de-multiplex STDOUT and STDERR with pseudo-tty option (-t
) or run python -i 2>&1
, the issue does not occur.
Is there any update on this issue?
no, I think @dperny is currently assigned on other tasks, but contributions are welcome
Just to keep the thread alive, I can still reproduce on a fresh 18.03 instance. :heart:
Let’s try resurrecting @dperny :innocent:
For whatever it's worth, I can now also reproduce on containerd
1.4.0 directly: :disappointed:
root@846a3a0c541f:/# ctr version
Client:
Version: v1.4.0
Revision: 09814d48d50816305a8e6c1a4ae3e2bcc4ba725a
Go version: go1.13.15
Server:
Version: v1.4.0
Revision: 09814d48d50816305a8e6c1a4ae3e2bcc4ba725a
UUID: 969f7235-0233-45cf-b268-b061f7438fff
root@846a3a0c541f:/# ctr run --rm docker.io/library/busybox:latest test awk 'BEGIN { print "1 (out)" > "/dev/stdout"; fflush("/dev/stdout"); print "2 (err)" > "/dev/stderr"; fflush("/dev/stderr"); print "3 (out)" > "/dev/stdout"; fflush("/dev/stdout"); print "4 (err)" > "/dev/stderr"; fflush("/dev/stderr"); exit }' 2>&1
1 (out)
3 (out)
2 (err)
4 (err)
(the output of that should be 1-4 in ascending order :disappointed:)
It seems like, at least in the case of docker, this is not completely unexpected since this is coming down a single stream, copied from two separate goroutines for each stdio stream.
Description
When outputting to stdout and stderr in quick succession without a TTY, log/output lines are consistently returned in the wrong order.
Steps to reproduce the issue:
I started by reproducing with
perl
, but realized I could do so with justawk
, which simplifies the testing and makes the explicit flushing more obvious (and makes it easier to embed in a test later, if we can figure out what's causing it and fix it):Describe the results you received:
Describe the results you expected:
(which is exactly what I get if I add
--tty
, but then Docker's only getting a single stream of all output rather than separatestdout
andstderr
, and I can't specify--tty
onRUN
in aDockerfile
, which is the place I see this issue in real life and it causes actual pain by lumping errors for a longRUN
line together at the beginning or end without all the normal output which is the context for what the error means)Additional information you deem important (e.g. issue happens only occasionally):
This is definitely similar to https://github.com/docker/docker/issues/26986 (possibly the same), but I have good reason to believe it's not due to buffering outside Docker (which is why that issue was closed), and appears instead to be due to Docker itself doing something odd (either doing its own buffering, or not storing timestamps with enough granularity in order to accurately recreate the proper line ordering). This is the reason why my example
awk
usesfflush
explicitly (to make sure there's no output buffering on theawk
/container side).Output of
docker version
:Output of
docker info
:Additional environment details (AWS, VirtualBox, physical, etc.):
Sometimes the output is even more exciting: