"possibly malicious path detected" during build

[thaJeztah]

As seen on https://github.com/moby/moby/pull/44079#issuecomment-1241021045

[2022-09-08T15:21:21.530Z]  > [crun 1/2] RUN --mount=type=cache,sharing=locked,id=moby-crun-aptlib,target=/var/lib/apt     --mount=type=cache,sharing=locked,id=moby-crun-aptcache,target=/var/cache/apt         apt-get update && apt-get install -y --no-install-recommends             autoconf             automake             build-essential             libcap-dev             libprotobuf-c-dev             libseccomp-dev             libsystemd-dev             libtool             libudev-dev             libyajl-dev             python3             ;:
[2022-09-08T15:21:21.530Z] #0 1.373 runc run failed: unable to start container process: error during container init: error mounting "/var/lib/docker/buildkit/executor/resolv.conf" to rootfs at "/etc/resolv.conf": possibly malicious path detected -- refusing to operate on /var/lib/docker/buildkit/executor/qj8f5s2o4ep3euuoz99hyp7a0/rootfs/etc/resolv.conf (deleted)

[2022-09-08T15:21:09.465Z] + docker version
[2022-09-08T15:21:09.465Z] Client: Docker Engine - Community
[2022-09-08T15:21:09.465Z]  Version:           22.06.0-beta.0
[2022-09-08T15:21:09.465Z]  API version:       1.42
[2022-09-08T15:21:09.465Z]  Go version:        go1.18.3
[2022-09-08T15:21:09.465Z]  Git commit:        3e9117b
[2022-09-08T15:21:09.465Z]  Built:             Fri Jun  3 17:55:46 2022
[2022-09-08T15:21:09.465Z]  OS/Arch:           linux/amd64
[2022-09-08T15:21:09.465Z]  Context:           default
[2022-09-08T15:21:09.465Z] 
[2022-09-08T15:21:09.465Z] Server: Docker Engine - Community
[2022-09-08T15:21:09.465Z]  Engine:
[2022-09-08T15:21:09.465Z]   Version:          22.06.0-beta.0
[2022-09-08T15:21:09.465Z]   API version:      1.42 (minimum version 1.12)
[2022-09-08T15:21:09.465Z]   Go version:       go1.18.3
[2022-09-08T15:21:09.465Z]   Git commit:       38633e7
[2022-09-08T15:21:09.465Z]   Built:            Fri Jun  3 17:55:46 2022
[2022-09-08T15:21:09.465Z]   OS/Arch:          linux/amd64
[2022-09-08T15:21:09.465Z]   Experimental:     true
[2022-09-08T15:21:09.465Z]  containerd:
[2022-09-08T15:21:09.465Z]   Version:          1.6.8
[2022-09-08T15:21:09.465Z]   GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
[2022-09-08T15:21:09.465Z]  runc:
[2022-09-08T15:21:09.465Z]   Version:          1.1.4
[2022-09-08T15:21:09.465Z]   GitCommit:        v1.1.4-0-g5fd4c4d
[2022-09-08T15:21:09.465Z]  docker-init:
[2022-09-08T15:21:09.465Z]   Version:          0.19.0
[2022-09-08T15:21:09.465Z]   GitCommit:        de40ad0

[2022-09-08T15:21:09.745Z] + docker info
[2022-09-08T15:21:09.745Z] Client:
[2022-09-08T15:21:09.745Z]  Context:    default
[2022-09-08T15:21:09.745Z]  Debug Mode: false
[2022-09-08T15:21:09.745Z]  Plugins:
[2022-09-08T15:21:09.745Z]   buildx: Docker Buildx (Docker Inc.)
[2022-09-08T15:21:09.745Z]     Version:  f500bf6
[2022-09-08T15:21:09.745Z]     Path:     /home/ubuntu/.docker/cli-plugins/docker-buildx
[2022-09-08T15:21:09.745Z]   compose: Docker Compose (Docker Inc.)
[2022-09-08T15:21:09.745Z]     Version:  v2.10.2
[2022-09-08T15:21:09.745Z]     Path:     /usr/libexec/docker/cli-plugins/docker-compose
[2022-09-08T15:21:09.745Z]   scan: Docker Scan (Docker Inc.)
[2022-09-08T15:21:09.745Z]     Version:  v0.17.0
[2022-09-08T15:21:09.745Z]     Path:     /usr/libexec/docker/cli-plugins/docker-scan
[2022-09-08T15:21:09.745Z] 
[2022-09-08T15:21:09.745Z] Server:
[2022-09-08T15:21:09.745Z]  Containers: 0
[2022-09-08T15:21:09.745Z]   Running: 0
[2022-09-08T15:21:09.745Z]   Paused: 0
[2022-09-08T15:21:09.745Z]   Stopped: 0
[2022-09-08T15:21:09.745Z]  Images: 2
[2022-09-08T15:21:09.745Z]  Server Version: 22.06.0-beta.0
[2022-09-08T15:21:09.745Z]  Storage Driver: overlay2
[2022-09-08T15:21:09.745Z]   Backing Filesystem: extfs
[2022-09-08T15:21:09.745Z]   Supports d_type: true
[2022-09-08T15:21:09.745Z]   Using metacopy: false
[2022-09-08T15:21:09.745Z]   Native Overlay Diff: true
[2022-09-08T15:21:09.745Z]   userxattr: false
[2022-09-08T15:21:09.745Z]  Logging Driver: json-file
[2022-09-08T15:21:09.745Z]  Cgroup Driver: cgroupfs
[2022-09-08T15:21:09.745Z]  Cgroup Version: 1
[2022-09-08T15:21:09.745Z]  Plugins:
[2022-09-08T15:21:09.745Z]   Volume: local
[2022-09-08T15:21:09.745Z]   Network: bridge host ipvlan macvlan null overlay
[2022-09-08T15:21:09.745Z]   Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
[2022-09-08T15:21:09.745Z]  Swarm: inactive
[2022-09-08T15:21:09.745Z]  Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
[2022-09-08T15:21:09.745Z]  Default Runtime: runc
[2022-09-08T15:21:09.745Z]  Init Binary: docker-init
[2022-09-08T15:21:09.745Z]  containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
[2022-09-08T15:21:09.745Z]  runc version: v1.1.4-0-g5fd4c4d
[2022-09-08T15:21:09.745Z]  init version: de40ad0
[2022-09-08T15:21:09.745Z]  Security Options:
[2022-09-08T15:21:09.745Z]   apparmor
[2022-09-08T15:21:09.745Z]   seccomp
[2022-09-08T15:21:09.745Z]    Profile: builtin
[2022-09-08T15:21:09.745Z]  Kernel Version: 5.4.0-1084-aws
[2022-09-08T15:21:09.745Z]  Operating System: Ubuntu 18.04.6 LTS
[2022-09-08T15:21:09.745Z]  OSType: linux
[2022-09-08T15:21:09.745Z]  Architecture: x86_64
[2022-09-08T15:21:09.745Z]  CPUs: 2
[2022-09-08T15:21:09.745Z]  Total Memory: 7.565GiB
[2022-09-08T15:21:09.745Z]  Name: ip-10-100-94-65
[2022-09-08T15:21:09.745Z]  ID: d579015a-308f-4cf8-9f89-dc1345c8f9aa
[2022-09-08T15:21:09.745Z]  Docker Root Dir: /var/lib/docker
[2022-09-08T15:21:09.745Z]  Debug Mode: false
[2022-09-08T15:21:09.745Z]  Registry: https://index.docker.io/v1/
[2022-09-08T15:21:09.745Z]  Labels:
[2022-09-08T15:21:09.745Z]  Experimental: true
[2022-09-08T15:21:09.745Z]  Insecure Registries:
[2022-09-08T15:21:09.745Z]   127.0.0.0/8
[2022-09-08T15:21:09.745Z]  Live Restore Enabled: true
[2022-09-08T15:21:09.745Z] 
[2022-09-08T15:21:09.745Z] WARNING: No swap limit support

[2022-09-08T15:21:10.069Z] + echo check-config.sh version: 33a3680e08d1007e72c3b3f1454f823d8e9948ee
[2022-09-08T15:21:10.069Z] check-config.sh version: 33a3680e08d1007e72c3b3f1454f823d8e9948ee
[2022-09-08T15:21:10.069Z] + curl -fsSL -o /home/ubuntu/workspace/moby_PR-44079/check-config.sh https://raw.githubusercontent.com/moby/moby/33a3680e08d1007e72c3b3f1454f823d8e9948ee/contrib/check-config.sh
[2022-09-08T15:21:10.069Z] + bash /home/ubuntu/workspace/moby_PR-44079/check-config.sh
[2022-09-08T15:21:10.069Z] warning: /proc/config.gz does not exist, searching other paths for kernel config ...
[2022-09-08T15:21:10.069Z] info: reading kernel config from /boot/config-5.4.0-1084-aws ...
[2022-09-08T15:21:10.069Z] 
[2022-09-08T15:21:10.069Z] Generally Necessary:
[2022-09-08T15:21:10.069Z] - cgroup hierarchy: properly mounted [/sys/fs/cgroup]
[2022-09-08T15:21:10.069Z] - apparmor: enabled and tools installed
[2022-09-08T15:21:10.069Z] - CONFIG_NAMESPACES: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_NET_NS: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_PID_NS: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_IPC_NS: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_UTS_NS: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_CGROUPS: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_CGROUP_CPUACCT: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_CGROUP_DEVICE: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_CGROUP_FREEZER: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_CGROUP_SCHED: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_CPUSETS: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_MEMCG: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_KEYS: enabled
[2022-09-08T15:21:10.069Z] - CONFIG_VETH: enabled (as module)
[2022-09-08T15:21:10.069Z] - CONFIG_BRIDGE: enabled (as module)
[2022-09-08T15:21:10.069Z] - CONFIG_BRIDGE_NETFILTER: enabled (as module)
[2022-09-08T15:21:10.069Z] - CONFIG_IP_NF_FILTER: enabled (as module)
[2022-09-08T15:21:10.069Z] - CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
[2022-09-08T15:21:10.069Z] - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
[2022-09-08T15:21:10.069Z] - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
[2022-09-08T15:21:10.069Z] - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
[2022-09-08T15:21:10.069Z] - CONFIG_NETFILTER_XT_MARK: enabled (as module)
[2022-09-08T15:21:10.069Z] - CONFIG_IP_NF_NAT: enabled (as module)
[2022-09-08T15:21:10.325Z] - CONFIG_NF_NAT: enabled (as module)
[2022-09-08T15:21:10.325Z] - CONFIG_POSIX_MQUEUE: enabled
[2022-09-08T15:21:10.325Z] - CONFIG_CGROUP_BPF: enabled
[2022-09-08T15:21:10.325Z] 
[2022-09-08T15:21:10.325Z] Optional Features:
[2022-09-08T15:21:10.325Z] - CONFIG_USER_NS: enabled
[2022-09-08T15:21:10.325Z] - CONFIG_SECCOMP: enabled
[2022-09-08T15:21:10.325Z] - CONFIG_SECCOMP_FILTER: enabled
[2022-09-08T15:21:10.325Z] - CONFIG_CGROUP_PIDS: enabled
[2022-09-08T15:21:10.325Z] - CONFIG_MEMCG_SWAP: enabled
[2022-09-08T15:21:10.325Z] - CONFIG_MEMCG_SWAP_ENABLED: missing
[2022-09-08T15:21:10.326Z]     (cgroup swap accounting is currently not enabled, you can enable it by setting boot option "swapaccount=1")
[2022-09-08T15:21:10.326Z] - CONFIG_BLK_CGROUP: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_BLK_DEV_THROTTLING: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_CGROUP_PERF: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_CGROUP_HUGETLB: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_NET_CLS_CGROUP: enabled (as module)
[2022-09-08T15:21:10.326Z] - CONFIG_CGROUP_NET_PRIO: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_CFS_BANDWIDTH: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_FAIR_GROUP_SCHED: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_RT_GROUP_SCHED: missing
[2022-09-08T15:21:10.326Z] - CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
[2022-09-08T15:21:10.326Z] - CONFIG_IP_VS: enabled (as module)
[2022-09-08T15:21:10.326Z] - CONFIG_IP_VS_NFCT: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_IP_VS_PROTO_TCP: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_IP_VS_PROTO_UDP: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_IP_VS_RR: enabled (as module)
[2022-09-08T15:21:10.326Z] - CONFIG_SECURITY_SELINUX: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_SECURITY_APPARMOR: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_EXT4_FS: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_EXT4_FS_POSIX_ACL: enabled
[2022-09-08T15:21:10.326Z] - CONFIG_EXT4_FS_SECURITY: enabled
[2022-09-08T15:21:10.326Z] - Network Drivers:
[2022-09-08T15:21:10.326Z]   - "overlay":
[2022-09-08T15:21:10.326Z]     - CONFIG_VXLAN: enabled (as module)
[2022-09-08T15:21:10.326Z]     - CONFIG_BRIDGE_VLAN_FILTERING: enabled
[2022-09-08T15:21:10.326Z]       Optional (for encrypted networks):
[2022-09-08T15:21:10.582Z]       - CONFIG_CRYPTO: enabled
[2022-09-08T15:21:10.582Z]       - CONFIG_CRYPTO_AEAD: enabled
[2022-09-08T15:21:10.582Z]       - CONFIG_CRYPTO_GCM: enabled
[2022-09-08T15:21:10.582Z]       - CONFIG_CRYPTO_SEQIV: enabled
[2022-09-08T15:21:10.582Z]       - CONFIG_CRYPTO_GHASH: enabled
[2022-09-08T15:21:10.582Z]       - CONFIG_XFRM: enabled
[2022-09-08T15:21:10.582Z]       - CONFIG_XFRM_USER: enabled (as module)
[2022-09-08T15:21:10.582Z]       - CONFIG_XFRM_ALGO: enabled (as module)
[2022-09-08T15:21:10.582Z]       - CONFIG_INET_ESP: enabled (as module)
[2022-09-08T15:21:10.582Z]   - "ipvlan":
[2022-09-08T15:21:10.582Z]     - CONFIG_IPVLAN: enabled (as module)
[2022-09-08T15:21:10.582Z]   - "macvlan":
[2022-09-08T15:21:10.582Z]     - CONFIG_MACVLAN: enabled (as module)
[2022-09-08T15:21:10.582Z]     - CONFIG_DUMMY: enabled (as module)
[2022-09-08T15:21:10.582Z]   - "ftp,tftp client in container":
[2022-09-08T15:21:10.582Z]     - CONFIG_NF_NAT_FTP: enabled (as module)
[2022-09-08T15:21:10.582Z]     - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
[2022-09-08T15:21:10.582Z]     - CONFIG_NF_NAT_TFTP: enabled (as module)
[2022-09-08T15:21:10.582Z]     - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
[2022-09-08T15:21:10.582Z] - Storage Drivers:
[2022-09-08T15:21:10.582Z]   - "aufs":
[2022-09-08T15:21:10.582Z]     - CONFIG_AUFS_FS: enabled (as module)
[2022-09-08T15:21:10.582Z]   - "btrfs":
[2022-09-08T15:21:10.582Z]     - CONFIG_BTRFS_FS: enabled (as module)
[2022-09-08T15:21:10.582Z]     - CONFIG_BTRFS_FS_POSIX_ACL: enabled
[2022-09-08T15:21:10.582Z]   - "devicemapper":
[2022-09-08T15:21:10.582Z]     - CONFIG_BLK_DEV_DM: enabled
[2022-09-08T15:21:10.582Z]     - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
[2022-09-08T15:21:10.582Z]   - "overlay":
[2022-09-08T15:21:10.582Z]     - CONFIG_OVERLAY_FS: enabled (as module)
[2022-09-08T15:21:10.582Z]   - "zfs":
[2022-09-08T15:21:10.582Z]     - /dev/zfs: present
[2022-09-08T15:21:10.582Z]     - zfs command: missing
[2022-09-08T15:21:10.582Z]     - zpool command: missing
[2022-09-08T15:21:10.582Z] 
[2022-09-08T15:21:10.582Z] Limits:
[2022-09-08T15:21:10.582Z] - /proc/sys/kernel/keys/root_maxkeys: 1000000
[2022-09-08T15:21:10.582Z] 
[2022-09-08T15:21:10.582Z] + true

[AkihiroSuda]

https://github.com/opencontainers/runc/blob/v1.1.4/libcontainer/utils/utils.go#L123-L129

    // Double-check the path is the one we expected.
    procfd := "/proc/self/fd/" + strconv.Itoa(int(fh.Fd()))
    if realpath, err := os.Readlink(procfd); err != nil {
        return fmt.Errorf("procfd verification failed: %w", err)
    } else if realpath != path {
        return fmt.Errorf("possibly malicious path detected -- refusing to operate on %s", realpath)
    }

Looks like os.Readlink("/proc/self/fd/<FD>") is returning "/var/lib/docker/buildkit/executor/qj8f5s2o4ep3euuoz99hyp7a0/rootfs/etc/resolv.conf (deleted)". A race condition?

[thaJeztah]

Yes, Tõnis expected it was a race condition indeed

[tonistiigi]

@AkihiroSuda Iiuc then something deletes resolv.conf when runc is in the middle of open and readlink.

Possible suspects are boot running again for same state directory https://github.com/moby/buildkit/blob/master/executor/runcexecutor/executor.go#L108

Something goes wrong in resolv.conf caching (and possibly rename gets called) https://github.com/moby/buildkit/blob/abde08a5531d809a395cf648a31bca932b009af0/executor/oci/resolvconf.go#L29 . This code is quite messy. I wonder if it is worth it at all and we should always create a new file.