Docker image consists of two large identical layers for `bun` and `bunx`

marvinruder commented 1 year ago

What version of Bun is running?

1.0.1+31aec4ebe325982fc0ef27498984b0ad9969162b

What platform is your computer?

Darwin 22.6.0 arm64 arm

What steps can reproduce the bug?

docker pull oven/bun:1.0.1

# For detailed analysis:
mkdir bun && cd bun
docker save oven/bun | tar -xz

# Then have a look at the largest files: ./blobs/sha256/{f96a...,7ce0..., 4704...} (all are gzipped tarballs)

What is the expected behavior?

The image is approximately 65 MB large and contains only two larger layers (and some more smaller ones): one is the debian-slim base image (around 30 MB), the other contains the bun binary (30–35 MB based on architecture).

What do you see instead?

The image is 95–100 MB large (depending on architecture).

Docker Hub:

DIGEST	OS/ARCH	COMPRESSED SIZE
456d511a28e6	linux/amd64	100.52 MB
7190627f958b	linux/arm64	94.72 MB

Docker Hub Image Details:

Step	Command	Size
1	`ADD file ... in /`	29.96 MB
2	`CMD ["bash"]`	0 B
3	`RUN /bin/sh -c groupadd bun`	3.98 KB
4	`COPY docker-entrypoint.sh /usr/local/bin # buildkit`	294 B
5	`COPY /usr/local/bin/bun /usr/local/bin # buildkit`	35.28 MB
6	`COPY /usr/local/bin/bunx /usr/local/bin # buildkit`	35.28 MB
7	`WORKDIR /home/bun/app`	140 B
8	`ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]`	0 B
9	`CMD ["/usr/local/bin/bun"]`	0 B

Output during pull:

Using default tag: latest
bdf51429e651: Download complete 
7190627f958b: Download complete 
056088840a56: Download complete 
4704d387fc37: Downloading [=========>                                         ]  6.291MB/34.63MB # This is /usr/local/bin/bunx
353472a8cc3f: Download complete 
c297c40960e3: Download complete 
f96ab1515704: Downloading [===========================================>       ]  26.21MB/30.06MB # This is debian-slim
7ce0fd8317b6: Downloading [=========>                                         ]  6.291MB/34.63MB # This is /usr/local/bin/bun
a99d480c3b22: Download complete

Additional information

In https://github.com/oven-sh/bun/blob/c3455c0ceee6bbe399781819a42fff6cf24792e2/dockerhub/Dockerfile-debian#L56 we see that bunx is a symbolic link pointing to bun. However, symbolic links are resolved by the Dockerfile COPY command (mentioned e.g. in https://github.com/moby/moby/issues/40449), so the resulting image contains the exact same file twice, in two different layers.

marvinruder commented 1 year ago

This can easily be fixed by moving the ln -s /usr/local/bin/bun /usr/local/bin/bunx command from the build image to the target image, replacing the COPY --from=build /usr/local/bin/bunx /usr/local/bin instruction.

However, the Dockerfile at https://github.com/oven-sh/bun/tree/main/dockerhub/Dockerfile-debian is many months old (and there are some others in the repository), so I am not sure whether it is actually used to generate the images published at Docker Hub.

Just let me know if you want me to create a PR based on it for this issue.

polarathene commented 1 year ago

I built the (unpublished) distroless variant which had the same issue, but it was easy to fix:

$ docker save local-bun-build | gzip -c | wc -c | numfmt --to iec
40M

$ docker images
REPOSITORY                  TAG                IMAGE ID       CREATED          SIZE
local-bun-build             latest             020acf83835e   7 minutes ago    110MB

# The 12.6MB layer is the distroless base image.
$ docker image history local-bun-build
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
020acf83835e   7 minutes ago   CMD ["/usr/local/bin/bun"]                      0B        buildkit.dockerfile.v0
<missing>      7 minutes ago   ENTRYPOINT ["/usr/local/bin/bun"]               0B        buildkit.dockerfile.v0
<missing>      7 minutes ago   WORKDIR /app                                    0B        buildkit.dockerfile.v0
<missing>      7 minutes ago   COPY /usr/local/bin/bun /usr/local/bin/bunx …   95MB      buildkit.dockerfile.v0
<missing>      N/A                                                             12.6MB
<missing>      N/A                                                             219kB
<missing>      N/A                                                             346B
<missing>      N/A                                                             497B
<missing>      N/A                                                             0B
<missing>      N/A                                                             64B
<missing>      N/A                                                             149B
<missing>      N/A                                                             1.93MB
<missing>      N/A                                                             29.4kB
<missing>      N/A                                                             270kB

All I did was combine the two COPY, which can take a list of sources and a final destination:

# List of sources to destination (final path):
COPY --from=build \
  /usr/local/bin/bun /usr/local/bin/bunx \
  /usr/local/bin

As you can see uncompressed size is 110MB, with roughly 40MiB compressed size (approximation as technically it should be each individual layer compressed, not all layers compressed into single archive). The compression approximation would also be the same without the change applied, but results in a 205MB uncompressed image.

For comparison to the current DockerHub (debian:11-slim) image (AMD64):

$ docker save oven/bun | gzip -c | wc -c | numfmt --to iec
97M

$ docker images
oven/bun                    latest             647ebb4444b7   3 days ago       271MB

$ docker image history oven/bun
docker image history oven/bun
IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
647ebb4444b7   3 days ago    CMD ["/usr/local/bin/bun"]                      0B        buildkit.dockerfile.v0
<missing>      3 days ago    ENTRYPOINT ["/usr/local/bin/docker-entrypoin…   0B        buildkit.dockerfile.v0
<missing>      3 days ago    WORKDIR /home/bun/app                           0B        buildkit.dockerfile.v0
<missing>      3 days ago    COPY /usr/local/bin/bunx /usr/local/bin # bu…   95MB      buildkit.dockerfile.v0
<missing>      3 days ago    COPY /usr/local/bin/bun /usr/local/bin # bui…   95MB      buildkit.dockerfile.v0
<missing>      3 days ago    COPY docker-entrypoint.sh /usr/local/bin # b…   171B      buildkit.dockerfile.v0
<missing>      3 days ago    RUN /bin/sh -c groupadd bun       --gid 1000…   332kB     buildkit.dockerfile.v0
<missing>      12 days ago   /bin/sh -c #(nop)  CMD ["bash"]                 0B
<missing>      12 days ago   /bin/sh -c #(nop) ADD file:cb5fcc80c057b356a…   80.5MB

Electroid commented 1 year ago

Thanks for flagging, changes will likely go live today or tomorrow:

❯ docker image ls
REPOSITORY   TAG             IMAGE ID       CREATED          SIZE
bun          debian          5527d6997b17   39 seconds ago   206MB
bun          debian-before   9b4dea30d19f   17 minutes ago   293MB

marvinruder commented 1 year ago

@polarathene I am quite surprised to see your COPY instruction with two files taking only the space of one Bun binary on your machine. Apparently this is not the case for the image that is pushed to Docker Hub, the compressed layer takes up approximately 70 MB, which is the size of two gzipped bun binaries:

Which then leads to the distroless images being larger than the alpine or even the debian-slim images, which is probably not what we want.

marvinruder commented 1 year ago

Also, are you able to run bunx in a distroless container at all? I cannot use a RUN instruction in a distroless Dockerfile since no /bin/sh exists, and running e.g. docker run --rm -it --entrypoint /usr/local/bin/bunx oven/bun:distroless eslint --version results in

error: SystemResources

----- bun meta -----
Bun v1.0.3 (25e69c71) Linux arm64 #1 SMP PREEMPT Thu Sep  7 07:48:47 UTC 2023
BunxCommand: 
Elapsed: 4ms | User: 0ms | Sys: 3ms
RSS: 33.55MB | Peak: 12.58MB | Commit: 33.55MB | Faults: 0
----- bun meta -----

Crash report saved to:
  ~/.bun-crash/v1.0.3-1695802194813.crash

Search GitHub issues https://bun.sh/issues or ask for #help in https://bun.sh/discord

while it works fine using the alpine container. A different error occurs when using bun x via the command docker run --rm -it oven/bun:distroless x eslint --version:

error: Failed to run "eslint" due to error FileNotFound

polarathene commented 1 year ago

Apparently this is not the case for the image that is pushed to Docker Hub, the compressed layer takes up approximately 70 MB, which is the size of two gzipped bun binaries:

This is the 2nd run of the command, but I did perform a fresh build without cache on the PR Dockerfile I added the COPY to:

$ docker buildx build -t bun-small .
[+] Building 1.9s (9/9) FINISHED                                                                                                      docker:default
 => [internal] load build definition from Dockerfile                                                                                            0.0s
 => => transferring dockerfile: 2.34kB                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                 0.0s
 => [internal] load metadata for gcr.io/distroless/base-nossl-debian11:latest                                                                   1.5s
 => [internal] load metadata for docker.io/library/debian:bullseye-slim                                                                         1.9s
 => [build 1/2] FROM docker.io/library/debian:bullseye-slim@sha256:c618be84fc82aa8ba203abbb07218410b0f5b3c7cb6b4e7248fda7785d4f9946             0.0s
 => [stage-1 1/2] FROM gcr.io/distroless/base-nossl-debian11@sha256:62f7fbe4d0880d52422814c143f607068f07ba588f62907b3fdd8867065b980a            0.0s
 => CACHED [build 2/2] RUN apt-get update -qq     && apt-get install -qq --no-install-recommends       ca-certificates       curl       dirmng  0.0s
 => CACHED [stage-1 2/2] COPY --from=build   /usr/local/bin/bun /usr/local/bin/bunx   /usr/local/bin                                            0.0s
 => exporting to image                                                                                                                          0.0s
 => => exporting layers                                                                                                                         0.0s
 => => writing image sha256:36fe89fcccb035c7d4f81fc12dab8eb9c4965f009b93efff9ae930b7749f1982                                                    0.0s
 => => naming to docker.io/library/bun-small                                                                                                    0.0s

$ docker pull oven/bun:distroless
$ docker images
REPOSITORY   TAG          IMAGE ID       CREATED         SIZE
bun-small    latest       36fe89fcccb0   6 minutes ago   110MB
oven/bun     distroless   f649763a2ca5   5 hours ago     205MB

$ docker save bun-small | gzip -c | wc -c | numfmt --to iec
40M
$ docker save oven/bun:distroless | gzip -c | wc -c | numfmt --to iec
73M

$ docker history bun-small
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
36fe89fcccb0   9 minutes ago   ENTRYPOINT ["/usr/local/bin/bun"]               0B        buildkit.dockerfile.v0
<missing>      9 minutes ago   COPY /usr/local/bin/bun /usr/local/bin/bunx …   94.9MB    buildkit.dockerfile.v0
<missing>      N/A                                                             12.6MB
<missing>      N/A                                                             219kB
<missing>      N/A                                                             346B
<missing>      N/A                                                             497B
<missing>      N/A                                                             0B
<missing>      N/A                                                             64B
<missing>      N/A                                                             149B
<missing>      N/A                                                             1.93MB
<missing>      N/A                                                             29.4kB
<missing>      N/A                                                             270kB

$ docker image history oven/bun:distroless
IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
f649763a2ca5   5 hours ago   ENTRYPOINT ["/usr/local/bin/bun"]               0B        buildkit.dockerfile.v0
<missing>      5 hours ago   COPY /usr/local/bin/bun /usr/local/bin/bunx …   190MB     buildkit.dockerfile.v0
<missing>      N/A                                                           12.6MB
<missing>      N/A                                                           219kB
<missing>      N/A                                                           346B
<missing>      N/A                                                           497B
<missing>      N/A                                                           0B
<missing>      N/A                                                           64B
<missing>      N/A                                                           149B
<missing>      N/A                                                           1.93MB
<missing>      N/A                                                           29.4kB
<missing>      N/A                                                           270kB

Not sure why they are different. I built it from WSL2 terminal (Ubuntu 20.04) with Docker Desktop installed, which is the advised way on Windows (normally I use a Linux host).

$ docker info
Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.4
...

Server:
...
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.123.1-microsoft-standard-WSL2
...

The Docker Engine and Buildx versions look the same to me as what the Github Action workflow was run with 🤷‍♂️

At first I thought maybe it might have been cache related, but that's not in use. Perhaps filesystem / storage driver in the CI differs and that affects it? Or just pushing the result to the registry modifies the resulting image efficiency?
There's COPY --link, but I doubt that would make a difference here.

polarathene commented 1 year ago

Also, are you able to run bunx in a distroless container at all? I cannot use a RUN instruction in a distroless Dockerfile since no /bin/sh exists, and running e.g. docker run --rm -it --entrypoint /usr/local/bin/bunx oven/bun:distroless eslint --version results in

Same output when running the DockerHub distroless image.

When running the local build, there's a different failure:

$ docker run --rm -it --entrypoint /usr/local/bin/bunx bun-small eslint --version
docker: Error response from daemon: failed to create task for container:
  failed to create shim task: OCI runtime create failed:
    runc create failed: unable to start container process:
      exec: "/usr/local/bin/bunx": stat /usr/local/bin/bunx: not a directory: unknown:
      Are you trying to mount a directory onto a file (or vice-versa)?
      Check if the specified host path exists and is the expected type.

marvinruder commented 1 year ago

Interesting, so it would appear your local build does not include a valid /usr/local/bin/bunx – perhaps your Docker builder handles symlinks differently, does not resolve them and attempts to include them as a regular file, maybe messes up the metadata in the process? That would explain the different layer sizes as well. COPYing a symlink from one stage to another without resolving is not a thing Docker usually supports (I referenced a related Moby issue above), that’s why I was surprised about your layer size initially.

polarathene commented 1 year ago

Resolved 😅

The COPY I added also failed the same for regular bun (I'm an idiot for not even testing the command prior to PR 😬 )

Instead of placing the files in the bin/ directory, they became bin 🙄

After adjusting that to COPY to bin/ same behaviour you have and the filesize is also large. This has already been fixed by @Electroid

https://github.com/oven-sh/bun/blob/4d2b442a33f2f0538c4655152d1798f40f84d03d/dockerhub/distroless/Dockerfile#L62-L65

marvinruder commented 1 year ago

Ah, that’s a classic!

But if bunx does not work without a distro anyway, we may just kick it out? Even if we manage to fix the issues in the future, I don’t see a advantage of having a distroless image that is larger than an image with a distro, and the functionality of bunx is still always accessible via bun x.

polarathene commented 1 year ago

But if bunx does not work without a distro anyway, we may just kick it out? Even if we manage to fix the issues in the future, I don’t see a advantage of having a distroless image that is larger than an image with a distro, and the functionality of bunx

Should work now if the use of RUN --mount is acceptable: https://github.com/oven-sh/bun/pull/6100

oven-sh / bun