Open RuRo opened 4 years ago
I might be wrong, but I can't imagine that this behavior would be hard to implement,
This actually seems impossible to implement. There are two modes of writing the progress: to the terminal and to the text stream. If we write to text stream then there is no way to inject anything in between the already written rows. So the only way is to block or to do (kind-of) smart switching like we do today. Otoh on tty mode we only have the screen coordinates to work with, we can replace the lines in current screen but we can't write anything outside of it or have content flowing.
I think the closest that you are asking and that would be useful would be to just keep the current tty mode but once the build is over and it reprints the summary, print all the logs as well and not hide them.
Also, some tips are to pipe the plain progress to sort if you don't care about it blocking. What people have asked before is to also make the vertex switching configurable so you can set ENV that define how much you want the stream to block or switch.
@tonistiigi I am not sure, that your assertion about how the tty
mode currently works is true (at least it appears to work differently to me). For example, consider that when the build process starts, the terminal is not cleared and the only line that appears below the shell prompt is
[+] Building 0.0s (0/2)
Then, as the build progresses, lines are constantly added after this line. Meanwhile this line is constantly updated to keep track of the time, ending with
[+] Building 19.5s (9/9) FINISHED
The same is true about the "scrolling" logs. At the beginning, the logs are only 1 line high and are then growing until they reach 6 lines. So clearly, the size of the "virtual" terminal is constantly changing throughout the build. I see no theoretical reason, why the logs couldn't just continue growing past the current 6 lines.
Unfortunately, your suggestion about sorting the output of --progress=plain
doesn't quite work.
1) Piping to sort
or even sort -n
/-h
doesn't work because the #
sign before the number forces alphabetical sorting, which puts #10
between #1
and #2
.
2) All the empty lines get sorted to the top.
3) The lines inside each numbered block also get sorted.
4) The blocks are not contiguous and are instead broken up by #123 ...
Theoretically, I could write some program, that would properly parse the plain
output and format it in the way I described in my original post, but it isn't really a feasible solution IMO as that would require me to ship a parser/printer program everywhere I want to build docker images, which is kind of ridiculous to be honest.
I really think, that having a workable human-readable output format is really the job of buildkit
. If buildkit
wants to eventually become the default docker build implementation, you will at least need to have a comparable quality of output and parallel builds are not a good enough excuse IMO.
are then growing until they reach 6 lines
The 6 limit is of course arbitrary. I'm just saying that it can't be ever-growing. The way it keeps the 6 line limit is monitoring your current screen size, finding out if there is space for the logs, finding the lines where to write the logs are on your screen, clearing these lines, and then rewriting with new content.
@tiborvass I think I saw you using sort
, did you set some special flags. Obviously this is just a workaround and not perfect.
Ah. After some experimenting with the current --progress=tty
I now understand what you mean. The current tty output mode is indeed dependent on the tty size. For example, if there are more lines in the output than the height of your terminal, then the output doesn't scroll, but instead gets truncated.
Damn, what a bummer. I am once again confronted by the fact that sh
/bash
/term
are such a huge load of horrible legacy nightmares. And this limitation really seems to be quite deep in the animal brain of VT100
. 😩
Hmm, okay. Can we think of some workaround/compromise involving partial "blocking/buffering" with --progress=plain
during parallel stages? The current breaking of blocks with ...
is really unusable IMO.
I guess, I am thinking of something like the current output format of docker build
, but in the event, that there are parallel build steps, the logs are printed in their order of appearance in the Dockerfile
.
I wonder if anyone was able to solve the mixed up stdout that github actions log streams when buildkit runs with --progress=plain
? it builds stages of dockerfile in parallel and writes logs in a mix as well, so it's impossible to understand what is what. is there a solution for that? (since i see this thread is still open, maybe a WIP?)
A similar issue was already raised in the past (#526), but it got closed after the
--progress
option was implemented. In my opinion, the current--progress
option conflates "tty support" and "output verbosity". As a result, there is currently no way to get a "verbose" human-readable output format.For example, running
docker build --no-cache --progress=tty
withthis Dockerfile
shows this log during the actual building process
but then collapses the output to this shorter log after the build is complete.
This behavior is undesirable, since most developers (in my experience) want to be able to look at the full logs during the build and after the build. The currently "recommended" way to get the full logs is to use
--progress=plain
.However, running
docker build --no-cache --progress=plain
with the sameDockerfile
results inthis unreadable mess, where the logs from different parallel stages are mixed together.
So in short,
--progress=tty
only shows the last 6 lines from each build and clears the logs after the build is done and--progress=plain
can't handle parallel builds properly and is overall actually quite ugly. In my opinion, we really need a--progress=verbose
option, which would act exactly like--progress=tty
, but the logs will not be limited to 6 lines and they will not get erased after the build step is complete.I might be wrong, but I can't imagine that this behavior would be hard to implement, since you just need to turn off the log "truncation" at 7 lines and after the build step finishes.