BuildKit builds (via Docker) are broken if /etc/hosts or /etc/resolv.conf is replaced

[rassie]

My Dockerfile:

from archlinux:20191105
run pacman -Sy --noconfirm filesystem

The filesystem package contains most of the base system files, as can be seen at https://www.archlinux.org/packages/core/x86_64/filesystem/. If it's upgraded, it tries to write /etc/hosts and /etc/resolv.conf, which leads to errors with BuildKit ("classical" Docker build is fine):

$ DOCKER_BUILDKIT=0 docker build .
Sending build context to Docker daemon  159.8MB
Step 1/2 : from archlinux:20191105
20191105: Pulling from library/archlinux
Digest: sha256:3fcb6f0c3a1266b579f7d5a89cbb66db1530e8dd533794b9c9588b630255b754
Status: Downloaded newer image for archlinux:20191105
 ---> 5ee688d008f4
Step 2/2 : run pacman -Sy --noconfirm filesystem
 ---> Running in f092dfd264b8
:: Synchronizing package databases...
downloading core.db...
downloading extra.db...
downloading community.db...
resolving dependencies...
looking for conflicting packages...

Packages (1) filesystem-2019.10-2

Total Download Size:   0.03 MiB
Total Installed Size:  0.04 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
downloading filesystem-2019.10-2-x86_64.pkg.tar.xz...
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Processing package changes...
upgrading filesystem...
warning: directory permissions differ on /srv/ftp/
filesystem: 755  package: 555
:: Running post-transaction hooks...
(1/4) Creating system user accounts...
(2/4) Applying kernel sysctl settings...
  Skipped: Current root is not booted.
(3/4) Creating temporary files...
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
(4/4) Arming ConditionNeedsUpdate...
Removing intermediate container f092dfd264b8
 ---> a7fa3f88205d
Successfully built a7fa3f88205d

docker build .
[+] Building 16.4s (5/5) FINISHED                                                                                                    
 => [internal] load build definition from Dockerfile                                                                            0.1s
 => => transferring dockerfile: 105B                                                                                            0.0s
 => [internal] load .dockerignore                                                                                               0.1s
 => => transferring context: 2B                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/archlinux:20191105                                                           0.5s
 => CACHED [1/2] FROM docker.io/library/archlinux:20191105@sha256:3fcb6f0c3a1266b579f7d5a89cbb66db1530e8dd533794b9c9588b630255  0.0s
 => ERROR [2/2] RUN pacman -Sy --noconfirm filesystem                                                                          15.7s
------                                                                                                                               
 > [2/2] RUN pacman -Sy --noconfirm filesystem:                                                                                      
#5 0.578 :: Synchronizing package databases...                                                                                       
#5 1.578 downloading core.db...                                                                                                      
#5 3.764 downloading extra.db...                                                                                                     
#5 7.726 downloading community.db...                                                                                                 
#5 14.71 resolving dependencies...
#5 14.72 looking for conflicting packages...
#5 14.72 
#5 14.72 Packages (1) filesystem-2019.10-2
#5 14.72 
#5 14.72 Total Download Size:   0.03 MiB
#5 14.72 Total Installed Size:  0.04 MiB
#5 14.72 Net Upgrade Size:      0.00 MiB
#5 14.72 
#5 14.72 :: Proceed with installation? [Y/n] 
#5 14.72 :: Retrieving packages...
#5 15.30 downloading filesystem-2019.10-2-x86_64.pkg.tar.xz...
#5 15.51 checking keyring...
#5 15.56 checking package integrity...
#5 15.59 loading package files...
#5 15.59 checking for file conflicts...
#5 15.59 checking available disk space...
#5 15.59 error: Partition /etc/resolv.conf is mounted read only
#5 15.59 error: Partition /etc/hosts is mounted read only
#5 15.59 error: not enough free disk space
#5 15.59 error: failed to commit transaction (not enough free disk space)
#5 15.60 Errors occurred, no packages were upgraded.
------

[tonistiigi]

What version of docker is this?

[rassie]

I can give you an exact version when I'm at the console again, but it's whatever the current version for Ubuntu is (Docker's repositories, not Ubuntu's). I've also reproduced the problem with a current -dind Docker container.

[rassie]

@tonistiigi my version is

Client:
 Debug Mode: false

Server:
 Containers: 7
  Running: 0
  Paused: 0
  Stopped: 7
 Images: 106
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.3.0-21-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.22GiB
[snip]
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

[tonistiigi]

Yes, this is somewhat expected by the current implementation. These files are configured by the container runtime. Even with the old implementation, these files do not really persist, you can write into them but the next command will not see the data you wrote, leading to a different error case. BuildKit will not allow writes instead of silently ignoring them. We could try to keep compatibility but it does have a bit of a performance overhead that would apply to all the containers, not just the ones the try to write to these files.

[ShemTovYosef]

I edited them to fake the docker hostname for applications that rely on these files and it helped and needed just during build stage due the limit to change the host. For running containers we can specify the exactly hostname and edit /etc/hosts without any limits so this feature is missing in buildx

[qualiaa]

I was having this issue, under the impression that --add-host was a runtime only flag -- it turns out I was wrong, and the solution for me was to use docker build --add-host=hostname:ip ... and remove modification of /etc/hosts from my Dockerfile. Not sure this will work for every use-case, though.

[ShemTovYosef]

Hi,

If you want to set specific hostname for during build time, now it implemented under this pull request: https://github.com/moby/buildkit/pull/1339 You can vote for it to be merged asap.

[jamshid]

Also running into this... Is there any way to allow DOCKER_BUILDKIT=1 docker build . with below Dockerfile to succeed, so a legacy package installation does not have to be rewritten?

It's okay if the /etc/hosts changes are thrown away (pre-DOCKER_BUILDKIT behavior), the package installer just can't get the Read-only file system error.

I tried RUN --mount=type=tmpfs ... but that needs to replace the entire /etc directory and I don't want to lose other changes made by the legacy package installation.

FROM centos:6.10 as intermediate

# A legacy package installation tries something like this, which fails with DOCKER_BUILDKIT=1
#     /bin/sh: /etc/hosts: Read-only file system
#
RUN echo 127.0.0.1 foo >> /etc/hosts

PS: this older issue is not really related: https://github.com/moby/moby/issues/11950

[tommyjcarpenter]

I am also running into this, just trying to build a simple archlinux container. Turning off buildkit solves this problem. It would appear recently (sometime, in 2020 maybe, between June and December?) Docker for Mac made a signfigant shift to build kit? This same Dockerfile that used to work now does not.

I posted this question before finding this thread, for full details: https://unix.stackexchange.com/questions/631177/arch-linux-in-docker-on-a-free-system-is-running-out-of-space/631178#631178

[tonistiigi]

I don't think there are plans to "fix" this. As I explained before these files are not modifiable in containers and managed by the runtime. Changing them leads to confusing behavior and broken images. --add-host works as intended also in BuildKit.

[tommyjcarpenter]

I understand. But I think what will happen is that people are going to hit this and it will be "obscure" to them because it's indirect. I'm not directly doing anything with resolve or hosts. I'm merely building the same docker image that I've built for years, in the same way, and now suddenly it doesn't work. The error that gets posted out, as I wrote in that stackexchange post highlights the error here, albeit with a minor red herring about space, but again to the user it's Indirect unless they are doing things (purposefully) with the hosts or resolve file. People running say just an OS like Arch or Ubuntu, this could be a "hidden" landmine in their typical work flow; especially since they may not be aware that common docker tools have recently switched to buildkit "under the covers" (docker for mac for example)

[thaJeztah]

I don't think there are plans to "fix" this. As I explained before these files are not modifiable in containers and managed by the runtime. Changing them leads to confusing behavior and broken images. --add-host works as intended also in BuildKit.

It's a tricky one; I agree that for users that want to customise during build, --add-host should be the recommendation; I guess (reading the discussion above) the problem, is that some packages (for whatever reason) appear to want to have these files writable. While it's worth looking why they want to do so, generally this would be out of control for the user that performs the build.

I saw https://github.com/moby/moby/issues/11950 was linked above; probably relevant here are https://github.com/moby/moby/issues/2267, and https://github.com/moby/moby/pull/5129, which (although not "persisting"), makes both of these writable by mounting the files.

I do share the concerns about performance though, so not sure what's best.

[binhex]

@thaJeztah sorry to labour the issue, but am i right in saying that there is still no workaround for the original issue, where installing arch linux package 'filesystem' causes a read only error due to bind mounts to the host?, i have gone through the links above but cannot see a way around this issue that relate to installation of a package that needs write access to /etc/hosts and /etc/resolv.conf please correct me if i'm wrong here.

[thaJeztah]

@binhex correct; BuildKit currently mounts these read-only;

With BuildKit:

DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain -<<EOF
FROM busybox
RUN mount | grep /etc
RUN cat /etc/hosts
EOF

#5 [2/2] RUN mount | grep /etc
#5 sha256:30dbb0d55d6b8a70aaf13d4655662c2ee23283d7fcbccceb70ff0a011adef4f1
#5 0.277 /dev/vda1 on /etc/resolv.conf type ext4 (ro,nosuid,nodev,noexec,relatime)
#5 0.277 /dev/vda1 on /etc/hosts type ext4 (ro,nosuid,nodev,noexec,relatime)
#5 DONE 0.4s

Without BuildKit:

DOCKER_BUILDKIT=0 docker build --no-cache --progress=plain -<<EOF
FROM busybox
RUN mount | grep /etc
EOF

Step 2/2 : RUN mount | grep /etc
 ---> Running in fbe072913354
/dev/vda1 on /etc/resolv.conf type ext4 (rw,relatime)
/dev/vda1 on /etc/hostname type ext4 (rw,relatime)
/dev/vda1 on /etc/hosts type ext4 (rw,relatime)

If your intent is to install the filesystem package to build a custom/more recent "base image" for arch linux, please refer to https://docs.docker.com/develop/develop-images/baseimages/

[binhex]

Just for anybody who finds this useful, i stumbled across the OP issue when using GitHub Actions and attempting to build a Arch Linux Docker image 'from scratch' using the buildx step:-

        uses: docker/setup-buildx-action@v1

if i then used the following 'uses' action then it blew up with the same read only issue for /etc/hostsand /etc/resolv.conf as OP.

        uses: docker/build-push-action@v2

the solution for me is simply to ignore the filesystem package during upgrade by doing the following:-

sed -i -e 's~#IgnorePkg.*~IgnorePkg = filesystem~g' '/etc/pacman.conf'

pacman then happily ignores the upgrade of filesystem and completes, this is obviously not ideal but it works and im happy enough to share this as a workaround for now until when/if the filesystem package stops attempting to modify the two files mentioned above.

[ian-h-chamberlain]

A lot of posts here mention --add-host as the proper way to change /etc/hosts, but what about /etc/resolv.conf? My organization uses different nameservers on the host than we do during build, so what we'd probably use ideally is the --dns option that docker run has. However there appears to be no such option for docker build, so for now using BuildKit doesn't seem to be an option for us.

Is there any way to explicitly set nameservers for a single docker build command?

[TingxinLi]

I don't think there are plans to "fix" this. As I explained before these files are not modifiable in containers and managed by the runtime. Changing them leads to confusing behavior and broken images. --add-host works as intended also in BuildKit.

I understand the situation to use --add-host for docker build, but how to use it for buildctl? I couldn't find any way to use --add-host in buildctl build

[tonistiigi]

@TingxinLi --opt add-hosts=name=ip,name2=ip2

[TingxinLi]

@TingxinLi --opt add-hosts=name=ip,name2=ip2

Thanks, I figured it out after reading the source code.

[mzihlmann]

how can i add-host to a bakefile?

edit: this does the trick for me, i can pass network=host when creating the builder (note: there's security implications for that)

docker buildx create --driver-opt network=host

[Miroka96]

I have to add to this issue as well... I either need /etc/hosts to be writable or need the hostname to not resolve to a loopback address. Preferably, the hostname in /etc/hosts should resolve to the primary IPv4 address of the container like it does in non-buildkit mode.

Reasoning: I have to prebuild an Oracle database during a container build, sigh... (With their current free database version 23.5, they don't accept loopback addresses as IPs for the given hostname and this hostname is taken from /etc/hostname, which is also read-only and correlates with the 127.0.0.1 entry in /etc/hosts. Adding the hostname with a different IP again using --add-host is ignored because it gets appended to the end of /etc/hosts. For now, I have to stick to the non-buildkit Docker build mode because it seems easier than getting a fix from Oracle...)