search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
736 stars 1.42k forks source link

[build] Experimental flag squash is removed with BuildKit causes image size to 2-4x larger #17319

Open sheiun-xu opened 11 months ago

sheiun-xu commented 11 months ago

Description

After docker version updated from 20 to 24 by commit e46be54 (#15652) the --squash experimental flag is no longer available in docker build.

The --squash flag is still invoked in slave.mk

And this would cause the docker-* images won't be squashed anymore, this will result in the final built image being 2-4x larger than the original.

The following warning message will be printed in any target/docker-*.gz.log

WARNING: experimental flag squash is removed with BuildKit. You should squash inside build using a multi-stage Dockerfile for efficiency.

Steps to reproduce the issue:

  1. make init
  2. make configure
  3. make target/docker-base-bullseye.gz
  4. docker load -i target/docker-base-bullseye.gz
  5. docker history docker-base-bullseye

Describe the results you received:

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
d378eeb0ad23        4 days ago          RUN /bin/sh -c rm /cache.tgz # buildkit         0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   3.13MB              buildkit.dockerfile.v0
<missing>           4 days ago          COPY etc/supervisor/containercfgd.conf /etc/…   231B                buildkit.dockerfile.v0
<missing>           4 days ago          COPY etc/supervisor/supervisord.conf /etc/su…   1.2kB               buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   1.46MB              buildkit.dockerfile.v0
<missing>           4 days ago          COPY root/.vimrc /root/.vimrc # buildkit        39B                 buildkit.dockerfile.v0
<missing>           4 days ago          COPY etc/rsyslog.d/* /etc/rsyslog.d/ # build…   303B                buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   453kB               buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   15.2MB              buildkit.dockerfile.v0
<missing>           4 days ago          COPY debs//socat_1.7.4.1-3_arm64.deb debs//l…   4.52MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   77.6MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   339kB               buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   4.69MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   1.36MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   163kB               buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   4.03MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   453kB               buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   13.7MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   3.04MB              buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   271MB               buildkit.dockerfile.v0
<missing>           4 days ago          COPY apt-multiple-retries /etc/apt/apt.conf.…   107B                buildkit.dockerfile.v0
<missing>           4 days ago          COPY no-check-valid-until /etc/apt/apt.conf.…   178B                buildkit.dockerfile.v0
<missing>           4 days ago          COPY no_install_recommend_suggest /etc/apt/a…   177B                buildkit.dockerfile.v0
<missing>           4 days ago          COPY sources.list.arm64 /etc/apt/sources.lis…   1.74kB              buildkit.dockerfile.v0
<missing>           4 days ago          COPY dpkg_01_drop /etc/dpkg/dpkg.cfg.d/01_dr…   998B                buildkit.dockerfile.v0
<missing>           4 days ago          ENV DEBIAN_FRONTEND=noninteractive              0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   1.19MB              buildkit.dockerfile.v0
<missing>           4 days ago          ENV DISTRO=bullseye                             0B                  buildkit.dockerfile.v0
<missing>           4 days ago          ENV IMAGENAME=docker-base-bullseye              0B                  buildkit.dockerfile.v0
<missing>           4 days ago          RUN |2 SONIC_VERSION_CACHE=cache SONIC_VERSI…   232kB               buildkit.dockerfile.v0
<missing>           4 days ago          COPY vcache/ /sonic/target/vcache/docker-bas…   0B                  buildkit.dockerfile.v0
<missing>           4 days ago          COPY buildinfo /usr/local/share/buildinfo # …   26.5kB              buildkit.dockerfile.v0
<missing>           4 days ago          ARG SONIC_VERSION_CONTROL_COMPONENTS            0B                  buildkit.dockerfile.v0
<missing>           4 days ago          ARG SONIC_VERSION_CACHE                         0B                  buildkit.dockerfile.v0
<missing>           3 years ago         /bin/sh -c #(nop) ADD file:27bc8fdb3559dece7…   4.45MB              
<missing>           3 years ago         /bin/sh -c #(nop)  ENV ARCH=arm64 UBUNTU_SUI…   0B                  
<missing>           3 years ago         /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B                  
<missing>           3 years ago         /bin/sh -c #(nop) ADD file:66cac2b0328037785…   115MB

Describe the results you expected:

This result is from the old version almost every layer is 0B and the latest layer is merged.

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
581f61cc09ca        About an hour ago   RUN /bin/sh -c rm /cache.tgz # buildkit         0B                  buildkit.dockerfile.v0
<missing>           About an hour ago                                                   145MB               merge sha256:dc34bff1633b292662fd50f90050dc8b906f6d8f209b4efbfda225972a4f8d67 to sha256:873181a41b991c9878c60e441108002df9a23ef0bef25ca2f3494ad399d2132d
<missing>           About an hour ago   /bin/sh -c #(nop)  LABEL com.azure.sonic.man…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c post…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c post…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:0cc107d9583abacc…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:07707f5a5ad9f4ca…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c ln /…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:72a3f85df6ef23ab…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:f9f37473474b73fa…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c apt-…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c dpkg…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY multi:67d474349da9908…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c apt-…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c mkdi…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c apt-…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pip3…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c apt-…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c apt-…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:1519c061413d88ea…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:064709ca31490f58…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:2ef539619e61ab99…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:2347580ff6afa417…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY file:aafbed21b2425f61…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop)  ENV DEBIAN_FRONTEND=nonin…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c whil…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c find…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c pre_…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop)  ENV DISTRO=bullseye          0B                  
<missing>           About an hour ago   /bin/sh -c #(nop)  ENV IMAGENAME=docker-base…   0B                  
<missing>           About an hour ago   |1 SONIC_VERSION_CACHE=cache /bin/sh -c dpkg…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY dir:1f0c9c3fcb4591a86…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop) COPY dir:3a644c568267c17e2…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop)  ARG SONIC_VERSION_CONTROL…   0B                  
<missing>           About an hour ago   /bin/sh -c #(nop)  ARG SONIC_VERSION_CACHE      0B                  
<missing>           7 days ago          /bin/sh -c #(nop)  CMD ["bash"]                 0B                  
<missing>           7 days ago          /bin/sh -c #(nop) ADD file:71543995e4d314b0c…   124MB

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

stepanblyschak commented 11 months ago

The increase of the size comes from the fact that intermediate layers contain temporary content used to build/install packages which then gets usually removed by apt-get purge, clean commands, etc. As a short term fix we could use 3rd party tools like - https://github.com/goldmann/docker-squash to squash an image. In the long run we could really leverage multi staged builds.

saiarcot895 commented 11 months ago

In rules/config, do you have SONIC_USE_DOCKER_BUILDKIT enabled? I believe if it's not enabled, then BuildKit should get explicitly disabled because of export DOCKER_BUILDKIT=0 in slave.mk.

sheiun-xu commented 11 months ago

In rules/config, do you have SONIC_USE_DOCKER_BUILDKIT enabled? I believe if it's not enabled, then BuildKit should get explicitly disabled because of export DOCKER_BUILDKIT=0 in slave.mk.

Both tried, and the layers are not squashed although we specify SONIC_USE_DOCKER_BUILDKIT=y.

In originally (docker-ce<=20) the docker build is combined with docker-ce-cli and the --squash works. And now after docker-ce upgraded to 23.0.0 the docker build will point to docker-buildx-plugin (alias of docker buildx build).

An as mentioned in

The --squash flag is already deprecated from buildx but still work in original docker build, but after upgrade the docker>=23 the buildx will replace orignal docker build and the squash broken.

k-v1 commented 8 months ago

@sheiun-xu This bug was fixed for master branch here: https://github.com/sonic-net/sonic-buildimage/pull/14405 Do you have this fix in your branch? But ofc we'll need to do something when we switch to the docker buildkit.