unlinkat directory not empty on commit (rootless)

dg424 commented 2 weeks ago

Issue Description

Getting the following error "randomly" when trying to build an image using the following command -> podman build --isolation chroot -t .

17:11:18 COMMIT foo:latest 17:11:18 --> 4434cee3fd5 17:11:18 Successfully tagged localhost/foo:latest 17:11:32 time="2024-06-14T21:11:30Z" level=error msg="error deleting build container \"dffdfc25f7f9f183eaca0c83ad95cd42daa5fbe0f33ec56cc8c525f3b0d5a98f\": 1 error occurred:\n\t* unlinkat /var/lib/containers/storage/vfs/dir/2157acd33ff63d42f27f2b14276c62bd0dea9d6d856849bad18da2220dfdf9e9: directory not empty\n\n\n" 17:11:32 Error: unlinkat /var/lib/containers/storage/vfs/dir/2157acd33ff63d42f27f2b14276c62bd0dea9d6d856849bad18da2220dfdf9e9: directory not empty

Any ideas/things to try etc ?

Steps to reproduce the issue

Unfortunately, the issue cannot be reproduced reliably and as stated in the description, it seems to occur randomly.

Describe the results you received

See description

Describe the results you expected

No error and a successful image build.

podman info output

bash-4.4# podman version
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user 
Client:       Podman Engine
Version:      4.4.1
API Version:  4.4.1
Go Version:   go1.19.10
Built:        Wed Oct  4 14:55:19 2023
OS/Arch:      linux/amd64

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Runs on an GCP compute engine instance

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

giuseppe commented 2 weeks ago

Additional environment details

Runs on an GCP compute engine instance

please provide more information on the environment. I see only one mapping is available. How was the user created? Is it a nested container?

podman info output

bash-4.4# podman version
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user 
Client:       Podman Engine
Version:      4.4.1
API Version:  4.4.1
Go Version:   go1.19.10
Built:        Wed Oct  4 14:55:19 2023
OS/Arch:      linux/amd64

podman version is not as helpful as the podman info output that is requested by the PR template. Can you please provide the podman info output?

dg424 commented 2 weeks ago

Hi @giuseppe,

Here is the output of podman info:

podman info --debug
time="2024-06-17T15:05:31Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.6-1.module+el8.8.0+1265+fa25dd7a.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: a88a21e8953a6243d5f369f61a342bcaf0630aa1'
  cpuUtilization:
    idlePercent: 84.2
    systemPercent: 2.37
    userPercent: 13.42
  cpus: 48
  distribution:
    distribution: '"rocky"'
    version: "8.8"
  eventLogger: file
  hostname: build-20240617150503150-l8s55-g8z2c
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 0
      size: 1
    - container_id: 1
      host_id: 1
      size: 4294967294
    uidmap:
    - container_id: 0
      host_id: 0
      size: 1
    - container_id: 1
      host_id: 1
      size: 4294967294
  kernel: 5.15.0-1050-gke
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 52360847360
  memTotal: 101331390464
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.1.4-1.module+el8.8.0+1265+fa25dd7a.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.4
      spec: 1.0.2-dev
      go: go1.19.4
      libseccomp: 2.5.2
  os: linux
  remoteSocket:
    path: /run/user/0/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_SYS_CHROOT,CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.module+el8.8.0+1265+fa25dd7a.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 1439h 54m 32.00s (Approximately 59.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /root/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 3168432029696
  graphRootUsed: 1942340562944
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1696431319
  BuiltTime: Wed Oct  4 14:55:19 2023
  GitCommit: ""
  GoVersion: go1.19.10
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1

giuseppe commented 1 week ago

I see the message:

time="2024-06-17T15:05:31Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"

In what environment are you running that command? Is it a nested container? Directly on the host?

Also this is the issue tracker for the upstream development, so please try with a newer version of Podman to see if the issue still persists

dg424 commented 1 week ago

We already tried with the latest version and still the same issue. In regards to the environment, this is a k8s pod running rootless docker daemon.

giuseppe commented 1 week ago

I've tried to reproduce a similar environment, running nested podman but I am not able to reproduce it yet.

Could you try to run podman inside podman (so no Docker involved) and see if that behaves in the same way for you? You can just use the podman image, e.g. podman run podman ...`

Could you share your Dockerfile?

containers / podman