[output clipped, log limit 1MiB reached] and then exit code 7

jessehui commented 3 years ago

Docker build on my local machine works fine. But when I use GitHub actions to build the docker image, I got error. First I see this log: [output clipped, log limit 1MiB reached], and after a few minutes, I got error code 7. After multiple times of retry, the results are the same.

The log snippet is here (for full log, check this out: https://github.com/occlum/occlum/runs/1573929172?check_suite_focus=true):

2020-12-18T00:48:25.5043289Z #10 272.4 ../include/libunwind-common.h:39:31: note: in expansion of macro 'UNW_PASTE'
2020-12-18T00:48:25.5044095Z #10 272.4  # define UNW_PREFIX UNW_PASTE(UNW_PASTE(_UL,UNW_TARGET),_)
2020-12-18T00:48:25.5044612Z #10 272.4                                ^~~~~~~~~
2020-12-18T00:48:25.5045321Z #10 272.4 ../include/libunwind-common.h:35:31: not
2020-12-18T00:48:25.5045945Z #10 272.4 [output clipped, log limit 1MiB reached]
2020-12-18T00:52:18.2647230Z #10 ERROR: executor failed running [/bin/sh -c git clone -b sgx_2.11_for_occlum https://github.com/occlum/linux-sgx &&     cd linux-sgx &&     ./compile_and_install.sh no_mitigation USE_OPT_LIBS=2 &&     echo 'source /opt/intel/sgxsdk/environment' >> /root/.bashrc &&     rm -rf /tmp/linux-sgx]: exit code: 7
2020-12-18T00:52:18.2680001Z ------
2020-12-18T00:52:18.2682323Z  > [stage-1  6/30] RUN git clone -b sgx_2.11_for_occlum https://github.com/occlum/linux-sgx &&     cd linux-sgx &&     ./compile_and_install.sh no_mitigation USE_OPT_LIBS=2 &&     echo 'source /opt/intel/sgxsdk/environment' >> /root/.bashrc &&     rm -rf /tmp/linux-sgx:
2020-12-18T00:52:18.2684280Z ------
2020-12-18T00:52:18.2686098Z failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c git clone -b sgx_2.11_for_occlum https://github.com/occlum/linux-sgx &&     cd linux-sgx &&     ./compile_and_install.sh no_mitigation USE_OPT_LIBS=2 &&     echo 'source /opt/intel/sgxsdk/environment' >> /root/.bashrc &&     rm -rf /tmp/linux-sgx]: exit code: 7
2020-12-18T00:52:24.8045851Z ##[error]buildx call failed with: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c git clone -b sgx_2.11_for_occlum https://github.com/occlum/linux-sgx &&     cd linux-sgx &&     ./compile_and_install.sh no_mitigation USE_OPT_LIBS=2 &&     echo 'source /opt/intel/sgxsdk/environment' >> /root/.bashrc &&     rm -rf /tmp/linux-sgx]: exit code: 7
2020-12-18T00:52:24.8230306Z Post job cleanup.

Github actions yaml: https://github.com/occlum/occlum/actions/runs/428064733/workflow dockerfile: https://github.com/occlum/occlum/blob/master/tools/docker/Dockerfile.centos8.1

jiayunwei commented 3 years ago

i met the problem too

leimao commented 3 years ago

Any way to increase the log limit cache size?

tonistiigi commented 3 years ago

clipping can be disabled with buildx create --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=-1 --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=-1 but this should have no effect on the outcome of the build. If it does then please post a reproducer.

Mr-K10 commented 3 years ago

I am facing the same issue, Can someone tell me how to stop logging or increase log limits from 1 MiB to at least 10MiB.

k-hashimoto commented 3 years ago

I have same issue. I want to how to stop logging or increase limits.

tonistiigi commented 3 years ago

increase log limits from 1 MiB to at least 10MiB.

for 10MB --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10485760

jessehui commented 3 years ago

For those who use buildx github actions for image building, just need extra with to prevent this:

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          driver-opts: env.BUILDKIT_STEP_LOG_MAX_SIZE=10485760

OmryAr commented 3 years ago

I'm struggling with this myself, running a buildkit enabled docker build from a linux command line - pulling hairs on how to pass those params... can anyone please point me to the right direction? (docker does not accept --driver-opt, and restarting dockerd with BUILDKIT_STEP_LOG_MAX_SIZE=-1 and BUILDKIT_STEP_LOG_MAX_SPEED=-1 environment vars doesn't have any effect).

jessehui commented 3 years ago

@OmryAr I believe these parameters are provided by buildx which means you should build image with docker buildx build. And these parameters are used when creating a buildx instance. For more info, please read the buildx doc.

OmryAr commented 3 years ago

Thanks @jessehui ! So after a long while of digging into the docs and experimenting - I've managed to create a buildx instance set with these configuration params. Something on the lines of:

docker buildx create --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=1000000,env.BUILDKIT_STEP_LOG_MAX_SPEED=100000000 --use

So now my revised question would be - how can I configure the default instance (without creating another one - which I wasn't intending to... and seems wasteful)?

When running docker buildx build, out-of-the-box, I am running the default instance, right?
How can I pass --driver-opts to the default instance?

yardenshoham commented 3 years ago

I'm using a regular docker build, how can I increase the log limit?

OmryAr commented 3 years ago

Hi @yardenshoham ! After researching this quite deeply, I can try to explain this:

At the time of this writing, log limits cannot be arbitrarily set using normal params to the docker build command.
To set log limits globally, you need set appropriate environment params for the dockerd service, and restart it for the configuration to take effect. Assuming a systemd based Linux-distro server, you'll need to add the following lines to /etc/sysconfig/docker: BUILDKIT_STEP_LOG_MAX_SIZE=10000000 BUILDKIT_STEP_LOG_MAX_SPEED=10000000
Alternatively you might create a builder instance and set it up with your desired options, as shown in my comment above.

HTH!

phdoerfler commented 3 years ago

for convenience, here's a bash one-liner: docker buildx build . --builder "$(docker buildx create --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10000000 --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=10000000)"

LukeIreland1 commented 3 years ago

clipping can be disabled with buildx create --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=-1 --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=-1 but this should have no effect on the outcome of the build. If it does then please post a reproducer.

[output clipped, log limit -1B reached]

myzcid commented 3 years ago

[output clipped, log limit 1MiB reached]

crazy-max commented 3 years ago

Fixed by https://github.com/moby/buildkit/pull/1934

xkszltl commented 3 years ago

The clipping issue has been creating big trouble for us for long. It doubles the work of debugging because none of the meaningful error can be captured within the limited size. All the error has to be repro by running Dockerfile commands manually within docker run. And any one-off errors (e.g. network issues/race conditions/environment specific issues/...) is simply not reproducible.

However, even when we try to raise the limit with env var for dockerd (BUILDKIT_STEP_LOG_MAX_SIZE and BUILDKIT_STEP_LOG_MAX_SPEED set to 1~2G via systemd), it will just hang in the middle of build like this:

The docker build will hang there with partial log for hours in this case.

zengqingfu1442 commented 3 years ago

docker 19.03.1 does not support buildx subcommand. But 20.10.1 supports.

root@f05c610dce6d:/# docker buildx 
docker: 'buildx' is not a docker command.
See 'docker --help'

Update, 20.10.1 does not support either. Because I execute the above command in a docker-in-docker container.

zaufi commented 3 years ago

Also hit by this crap with normal docker build :(

Even --progress=plain doesn't help... This is insane!! I just can't get my full log and see the compiler error!

TomBonnerAtDerivitec commented 3 years ago

This is a fairly sad regression. Unable to view more than 1MB of logs on our own machine?

Here's my stats:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 9
  Running: 1
  Paused: 0
  Stopped: 8
 Images: 25
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.47-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.64GiB
 Name: docker-desktop

JJ commented 2 years ago

The question here, is the error precisely that the max log size has been reached, or is it hiding some other error?

maffei2443 commented 2 years ago

The true question is: how is this problem supposed to be solved? There must be a way to solve that, right? Maybe extending the max log size or even setting it to "unlimited".

abdennour commented 2 years ago

[output clipped, log limit 1MiB reached]

Happen for us when there is no build cache, and the app requires a mess of dependencies ( jar files). The log shows the downloaded dependencies & its nested dependencies until seeing that error:

[output clipped, log limit 1MiB reached]

We are using docker-compose to build using buildkit feature. Any thoughts how to fix it at the level of docker daemon ?

OmryAr commented 2 years ago

@abdennour @maffei2443 Please go early in this thread (around Jan-April 2021) for ways to solve this. It might have been addressed in a more elegant way in a later Docker release - so consider that too.

xkszltl commented 2 years ago

@OmryAr So far I haven't seen any good resolution in docker. Adding env var to docker daemon via systemd doesn't help much as that somehow introduced a random hanging issue for large log, even when it's only a few MB. We never had this issue until the release of this feature, or without setting those env var.

And there's no "unlimited" value.

I think a reasonable fix is to disable clipping by default, or at least have a reliable kill switch.

bumerang07 commented 2 years ago

I have the docker 20.10.11 And it seems that the only working solution is to add this variable DOCKER_BUILDKIT=0 before running docker:

DOCKER_BUILDKIT=0 docker build

This will prevent using new build functionalities

celetium commented 2 years ago

I have the docker 20.10.11 And it seems that the only working solution is to add this variable DOCKER_BUILDKIT=0 before running docker:

DOCKER_BUILDKIT=0 docker build

This will prevent using new build functionalities

This really work! thanks!

Doug910 commented 2 years ago

This worked on Windows I added DOCKER_BUILDKIT with value 0 to the system environment variables, then ran Docker Desktop 4.12.0, when I do a build the log keeps going, running all the way with no limit or hang ups.

okanbeydanol commented 2 years ago

This worked on Windows I added DOCKER_BUILDKIT with value 0 to the system environment variables, then ran Docker Desktop 4.12.0, when I do a build the log keeps going, running all the way with no limit or hang ups.

Its working on mac too.

ikovalyov commented 9 months ago

This is still an issue with docker-compose. Seems to be no way to disable this log limit there.

malaterre commented 7 months ago

I have the docker 20.10.11 And it seems that the only working solution is to add this variable DOCKER_BUILDKIT=0 before running docker:

DOCKER_BUILDKIT=0 docker build

This will prevent using new build functionalities

Works for me ... but if you are running inside gitlab you'll eventually reach:

Job's log exceeded limit of 4194304 bytes. Job execution will continue but no more output will be collected.

https://stackoverflow.com/questions/53541000/gitlab-ci-cd-jobs-log-exceeded-limit

prjemian commented 6 months ago

Today, with DOCKER_BUILDKIT=0 docker build and Docker version 26.0.0, build 2ae903e on linux, this report:

# started:  2024-04-02T12:24:32-05:00
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  29.18kB

docker / buildx

[output clipped, log limit 1MiB reached] and then exit code 7 #484