containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.86k stars 2.42k forks source link

Rootless podman on CircleCI fails to even build with error running container: from /usr/bin/crun: sd-bus call: Permission denied #16529

Closed adelton closed 1 year ago

adelton commented 2 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

I select bug here but this is more a question about what the assumptions are about the setup and what could be causing the crun / sd-dbus Permission deniced / Interactive authentication required failure or what configuration change to try.

Description

I try to add testing https://github.com/freeipa/freeipa-container on CircleCI on their Ubuntu 22.04 VMs.

Compared to GitHub Actions Ubuntu 22.04 where this works without issues once a session gets created by ssh to self, on CircleCI where loginctl shows that we already have session I'm hitting

STEP 2/2: RUN date
error running container: error from /usr/bin/crun creating container for [/bin/sh -c date]: sd-bus call: Permission denied
: exit status 1
Error: error building at STEP "RUN date": error while running runtime: exit status 1
Exited with code exit status 125

or with podman from download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/

STEP 2/2: RUN date
error running container: from /usr/bin/crun creating container for [/bin/sh -c date]: sd-bus call: Interactive authentication required.: Permission denied
: exit status 1
ERRO[0003] did not get container create message from subprocess: EOF 
Error: building at STEP "RUN date": while running runtime: exit status 1
Exited with code exit status 1

Steps to reproduce the issue:

  1. Have .circleci/config.yml in your project's repo with content to install podman and build an image with it in rootless fashion
    
    version: 2.1

jobs: build-image: machine: image: ubuntu-2204:edge resource_class: arm.medium steps:

workflows: test-podman: jobs:

Describe the results you received:

https://app.circleci.com/pipelines/github/adelton/freeipa-container/74/workflows/9a11b1d9-4040-4d9f-a005-e50532d4bb44/jobs/503

podman build -f Dockerfile.circleci .

STEP 1/2: FROM quay.io/centos/centos:stream9
Trying to pull quay.io/centos/centos:stream9...
Getting image source signatures
…
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN date
error running container: error from /usr/bin/crun creating container for [/bin/sh -c date]: sd-bus call: Permission denied
: exit status 1
Error: error building at STEP "RUN date": error while running runtime: exit status 1
Exited with code exit status 125

Describe the results you expected:

No error, image built.

Additional information you deem important (e.g. issue happens only occasionally):

The loginctl shows we are running in a session:

SESSION  UID USER     SEAT TTY
     c1 1001 circleci      

1 sessions listed.

I also tried systemd-run --scope --user and ssh to self to no avail. It seems like we have session alright, it just needs some permissions somewhere.

I get this on the CircleCI Ubuntu VMs in deterministic fashion. When I change the

      - run: sudo apt-get -y update
      - run: sudo apt-get -y install podman

steps to

      - run: sudo mkdir -p /etc/apt/keyrings
      - run: curl -fsSL https://download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/xUbuntu_$(lsb_release -rs)/Release.key
             | gpg --dearmor
             | sudo tee /etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg > /dev/null
      - run: echo
             "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/devel_kubic_libcontainers_unstable.gpg]
             https://download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/xUbuntu_$(lsb_release -rs)/ /"
             | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list > /dev/null
      - run: sudo apt-get update -y
      - run: sudo apt-get install -y podman skopeo

to test with the latest podman per https://podman.io/getting-started/installation#ubuntu, the output is slightly different but still error:

https://app.circleci.com/pipelines/github/adelton/freeipa-container/73/workflows/a086e653-cd68-47f9-851c-54a79d849c7f/jobs/502

podman build -f Dockerfile.circleci .

STEP 1/2: FROM quay.io/centos/centos:stream9
Trying to pull quay.io/centos/centos:stream9...
Getting image source signatures
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN date
error running container: from /usr/bin/crun creating container for [/bin/sh -c date]: sd-bus call: Interactive authentication required.: Permission denied
: exit status 1
ERRO[0003] did not get container create message from subprocess: EOF 
Error: building at STEP "RUN date": while running runtime: exit status 1
Exited with code exit status 1

Output of podman version:

Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.3
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/arm64
WARN[0000] Failed to add pause process to systemd sandbox cgroup: dbus: invalid bus address (no transport) 

Output of podman info:

host:
  arch: arm64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 2
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: ip-172-28-21-69
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.15.0-1022-aws
  linkmode: dynamic
  logDriver: journald
  memFree: 6990278656
  memTotal: 8116731904
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 0
  swapTotal: 0
  uptime: 1m 59.56s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/circleci/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/circleci/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1001/containers
  volumePath: /home/circleci/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.3
  OsArch: linux/arm64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

Listing... Done
podman/jammy,now 3.4.4+ds1-1ubuntu1 arm64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes; No

Additional environment details (AWS, VirtualBox, physical, etc.):

This is on CircleCI Ubuntu 22.04 ARM VM.

Luap99 commented 2 years ago

Sounds like you are missing the systemd user session. Can you try with the latest version?

adelton commented 2 years ago

When I added loginctl session-status, it shows

c1 - circleci (1001)
       Since: Wed 2022-11-16 22:30:24 UTC; 32s ago
      Leader: 916 (sudo)
      Remote: user root
     Service: sudo-i; type unspecified; class background
       State: active
        Unit: session-c1.scope
          ├─ 916 sudo -niHu circleci -- /tmp/circleci-machine-agent2515740742/circleci-agent _internal agent-runner --config -
          ├─ 925 /tmp/circleci-machine-agent2515740742/circleci-agent _internal agent-runner --config -
          ├─3732 podman
          ├─4772 /bin/bash -eo pipefail -c "loginctl session-status | cat"
          ├─5095 loginctl session-status
          └─5096 cat

Nov 16 22:30:27 ip-172-28-2-234 sudo[1289]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
Nov 16 22:30:35 ip-172-28-2-234 sudo[1289]: pam_unix(sudo:session): session closed for user root
Nov 16 22:30:36 ip-172-28-2-234 sudo[2278]: circleci : TTY=pts/0 ; PWD=/home/circleci/project ; USER=root ; COMMAND=/usr/bin/apt-get -y install podman
Nov 16 22:30:36 ip-172-28-2-234 sudo[2278]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1001)
Nov 16 22:30:43 ip-172-28-2-234 useradd[2840]: new user: name=dnsmasq, UID=115, GID=65534, home=/var/lib/misc, shell=/usr/sbin/nologin, from=/dev/pts/2
Nov 16 22:30:43 ip-172-28-2-234 usermod[2847]: change user 'dnsmasq' password
Nov 16 22:30:43 ip-172-28-2-234 chage[2854]: changed password expiry for dnsmasq
Nov 16 22:30:43 ip-172-28-2-234 chfn[2858]: changed user 'dnsmasq' information
Nov 16 22:30:53 ip-172-28-2-234 sudo[2278]: pam_unix(sudo:session): session closed for user root
Nov 16 22:30:54 ip-172-28-2-234 podman[3728]: 2022-11-16 22:30:54.002908492 +0000 UTC m=+0.163240424 system refresh

-- see https://app.circleci.com/pipelines/github/adelton/freeipa-container/98/workflows/9301ab35-a5fd-48a5-9d90-c4090232a2d1/jobs/505. So there is some user session there. Anything specific about it that I should look for?

What latest version do you have in mind? I install podman from download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/ in this case.

Luap99 commented 2 years ago

sudo or su will not create a proper systemd user session, see https://github.com/containers/podman/blob/main/troubleshooting.md#31-podman-run-fails-with-erro0000-xdg_runtime_dir-directory-runuser0-is-not-owned-by-the-current-user-or-error-creating-tmpdir-mkdir-runuser1000-permission-denied for alternatives.

I don't know much about the packaging but I assume the kubic repo has a more up to date version so I would try that one. I think it is very likely that this is already fixed in newer versions. I remember seeing issues like that before.

adelton commented 2 years ago

As I already mentioned above:

I also tried systemd-run --scope --user and ssh to self to no avail. It seems like we have session alright, it just needs some permissions somewhere.

I assume CircleCI falls into the category of "badly configured systemd session" at https://github.com/containers/podman/blob/main/troubleshooting.md#31-podman-run-fails-with-erro0000-xdg_runtime_dir-directory-runuser0-is-not-owned-by-the-current-user-or-error-creating-tmpdir-mkdir-runuser1000-permission-denied -- but what specifically should I be looking for? What sd-bus operations does podman do?

What latest version do you have in mind? I install podman from download.opensuse.org/repositories/devel:kubic:libcontainers:unstable/ in this case.

What kubic repo do you have in mind, different from this unstable one I've been trying it with?

Luap99 commented 2 years ago

Yes I think this is the correct kubic repo.

Podman doesn't do anything with sd-bus here. It is crun which is failing.

/usr/bin/crun creating container for [/bin/sh -c date]: sd-bus call: Permission denied
/usr/bin/crun creating container for [/bin/sh -c date]: sd-bus call: Interactive authentication required.: Permission denied

cc @giuseppe

@adelton Does a normal podman run work? You could also try installing runc and using this as your runtime for testing.

adelton commented 2 years ago

What do you mean exactly by normal podman? I thought that my use of podman was quite normal. :-)

When I installed runc and added --log-level=debug, I see

DEBU[0003] setting image creation date to 2022-11-11 16:00:36.341508704 +0000 UTC 
DEBU[0003] created new image ID "e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" 
DEBU[0003] saved image metadata "{\"signatures-sizes\":{\"sha256:7e593e51341bce240b4d0893e9d0e508b810ab84176fedd9027c4e34636e910b\":[]}}" 
DEBU[0003] set names of image "e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" to [quay.io/centos/centos:stream9] 
DEBU[0003] Pulled candidate quay.io/centos/centos:stream9 successfully 
DEBU[0003] Looking up image "e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" in local containers storage 
DEBU[0003] Trying "e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" ... 
DEBU[0003] parsed reference into "[overlay@/home/circleci/.local/share/containers/storage+/run/user/1001/containers]@e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" 
DEBU[0003] Found image "e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" as "e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" in local containers storage 
DEBU[0003] Found image "e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" as "e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" in local containers storage ([overlay@/home/circleci/.local/share/containers/storage+/run/user/1001/containers]@e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167) 
DEBU[0003] exporting opaque data as blob "sha256:e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" 
DEBU[0003] exporting opaque data as blob "sha256:e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" 
DEBU[0003] [graphdriver] trying provided driver "overlay" 
DEBU[0003] cached value indicated that overlay is supported 
DEBU[0003] cached value indicated that metacopy is not being used 
DEBU[0003] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0003] overlay test mount indicated that volatile is being used 
DEBU[0003] overlay: mount_data=,lowerdir=/home/circleci/.local/share/containers/storage/overlay/l/XWGI7OQW2Y6VRWIMSUNKVYFN4S,upperdir=/home/circleci/.local/share/containers/storage/overlay/de22f85c78cf33119a12c2c8d158584d71c35917b774059a40805a4a3da77047/diff,workdir=/home/circleci/.local/share/containers/storage/overlay/de22f85c78cf33119a12c2c8d158584d71c35917b774059a40805a4a3da77047/work,userxattr,volatile 
DEBU[0003] Container ID: 4b329f72580b21c1afb45e93ad40b591919dd2fe476c3423278ac242468d798a 
DEBU[0003] Parsed Step: {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[date] Flags:[] Attrs:map[] Message:RUN date Original:RUN date} 
STEP 2/2: RUN date
DEBU[0003] parsed reference into "[overlay@/home/circleci/.local/share/containers/storage+/run/user/1001/containers]@e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" 
DEBU[0003] exporting opaque data as blob "sha256:e08717ee1d9523552549e565e6cae5fadee41e089594db802f02dfd1ae115167" 
DEBU[0003] RUN imagebuilder.Run{Shell:true, Args:[]string{"date"}, Mounts:[]string(nil)}, docker.Config{Hostname:"", Domainname:"", User:"", Memory:0, MemorySwap:0, MemoryReservation:0, KernelMemory:0, CPUShares:0, CPUSet:"", PortSpecs:[]string(nil), ExposedPorts:map[docker.Port]struct {}{}, PublishService:"", StopSignal:"", StopTimeout:0, Env:[]string{"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"}, Cmd:[]string{"/bin/bash"}, Shell:[]string{}, Healthcheck:(*docker.HealthConfig)(nil), DNS:[]string(nil), Image:"", Volumes:map[string]struct {}{}, VolumeDriver:"", WorkingDir:"", MacAddress:"", Entrypoint:[]string{}, SecurityOpts:[]string(nil), OnBuild:[]string{}, Mounts:[]docker.Mount(nil), Labels:map[string]string{"io.buildah.version":"1.19.8", "org.label-schema.build-date":"20221111", "org.label-schema.license":"GPLv2", "org.label-schema.name":"CentOS Stream 9 Base Image", "org.label-schema.schema-version":"1.0", "org.label-schema.vendor":"CentOS"}, AttachStdin:false, AttachStdout:false, AttachStderr:false, ArgsEscaped:false, Tty:false, OpenStdin:false, StdinOnce:false, NetworkDisabled:false, VolumesFrom:""} 
DEBU[0003] using "/var/tmp/buildah1094495047" to hold bundle data 
DEBU[0003] Forcing use of an IPC namespace.             
DEBU[0003] Forcing use of a PID namespace.              
DEBU[0003] Forcing use of a user namespace.             
DEBU[0003] Resources: &define.CommonBuildOptions{AddHost:[]string{}, CgroupParent:"", CPUPeriod:0x0, CPUQuota:0, CPUShares:0x0, CPUSetCPUs:"", CPUSetMems:"", HTTPProxy:true, Memory:0, DNSSearch:[]string{}, DNSServers:[]string{}, DNSOptions:[]string{}, MemorySwap:0, LabelOpts:[]string(nil), OmitTimestamp:false, SeccompProfilePath:"/usr/share/containers/seccomp.json", ApparmorProfile:"", ShmSize:"65536k", Ulimit:[]string{}, Volumes:[]string{}, Secrets:[]string{}, SSHSources:[]string{}} 
DEBU[0004] /etc/system-fips does not exist on host, not mounting FIPS mode subscription 
DEBU[0000] bind mounted "/home/circleci/.local/share/containers/storage/overlay/de22f85c78cf33119a12c2c8d158584d71c35917b774059a40805a4a3da77047/merged" to "/var/tmp/buildah1094495047/mnt/rootfs" 
DEBU[0000] config = {"ociVersion":"1.0.2-dev","process":{"user":{"uid":0,"gid":0},"args":["/bin/sh","-c","date"],"env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME=4b329f72580b"],"cwd":"/","capabilities":{"bounding":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_NET_BIND_SERVICE","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"effective":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_NET_BIND_SERVICE","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"inheritable":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_NET_BIND_SERVICE","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"permitted":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_NET_BIND_SERVICE","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"],"ambient":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FOWNER","CAP_FSETID","CAP_KILL","CAP_NET_BIND_SERVICE","CAP_SETFCAP","CAP_SETGID","CAP_SETPCAP","CAP_SETUID","CAP_SYS_CHROOT"]},"rlimits":[{"type":"RLIMIT_NOFILE","hard":1024,"soft":1024}]},"root":{"path":"/var/tmp/buildah1094495047/mnt/rootfs"},"hostname":"4b329f72580b","mounts":[{"destination":"/dev","type":"tmpfs","source":"/dev","options":["private","strictatime","noexec","nosuid","mode=755","size=65536k"]},{"destination":"/dev/mqueue","type":"mqueue","source":"mqueue","options":["private","nodev","noexec","nosuid"]},{"destination":"/dev/pts","type":"devpts","source":"pts","options":["private","noexec","nosuid","newinstance","ptmxmode=0666","mode=0620"]},{"destination":"/dev/shm","type":"tmpfs","source":"shm","options":["private","nodev","noexec","nosuid","mode=1777","size=65536k"]},{"destination":"/proc","type":"proc","source":"/proc","options":["private","nodev","noexec","nosuid"]},{"destination":"/sys","type":"bind","source":"/sys","options":["rbind","private","nodev","noexec","nosuid","ro"]},{"destination":"/etc/hosts","type":"bind","source":"/var/tmp/buildah1094495047/hosts","options":["rbind"]},{"destination":"/etc/resolv.conf","type":"bind","source":"/var/tmp/buildah1094495047/resolv.conf","options":["rbind"]},{"destination":"/run/.containerenv","type":"bind","source":"/var/tmp/buildah1094495047/run/.containerenv","options":["rbind"]}],"linux":{"uidMappings":[{"containerID":0,"hostID":0,"size":1},{"containerID":1,"hostID":1,"size":65536}],"gidMappings":[{"containerID":0,"hostID":0,"size":1},{"containerID":1,"hostID":1,"size":65536}],"namespaces":[{"type":"pid"},{"type":"ipc"},{"type":"uts"},{"type":"mount"},{"type":"user"}],"seccomp":{"defaultAction":"SCMP_ACT_ERRNO","defaultErrnoRet":38,"architectures":["SCMP_ARCH_AARCH64","SCMP_ARCH_ARM"],"syscalls":[{"names":["bdflush","io_pgetevents","kexec_file_load","kexec_load","migrate_pages","move_pages","nfsservctl","nice","oldfstat","oldlstat","oldolduname","oldstat","olduname","pciconfig_iobase","pciconfig_read","pciconfig_write","sgetmask","ssetmask","swapcontext","swapoff","swapon","sysfs","uselib","userfaultfd","ustat","vm86","vm86old","vmsplice"],"action":"SCMP_ACT_ERRNO","errnoRet":1},{"names":["_llseek","_newselect","accept","accept4","access","adjtimex","alarm","bind","brk","capget","capset","chdir","chmod","chown","chown32","clock_adjtime","clock_adjtime64","clock_getres","clock_getres_time64","clock_gettime","clock_gettime64","clock_nanosleep","clock_nanosleep_time64","clone","clone3","close","close_range","connect","copy_file_range","creat","dup","dup2","dup3","epoll_create","epoll_create1","epoll_ctl","epoll_ctl_old","epoll_pwait","epoll_pwait2","epoll_wait","epoll_wait_old","eventfd","eventfd2","execve","execveat","exit","exit_group","faccessat","faccessat2","fadvise64","fadvise64_64","fallocate","fanotify_mark","fchdir","fchmod","fchmodat","fchown","fchown32","fchownat","fcntl","fcntl64","fdatasync","fgetxattr","flistxattr","flock","fork","fremovexattr","fsconfig","fsetxattr","fsmount","fsopen","fspick","fstat","fstat64","fstatat64","fstatfs","fstatfs64","fsync","ftruncate","ftruncate64","futex","futex_time64","futimesat","get_robust_list","get_thread_area","getcpu","getcwd","getdents","getdents64","getegid","getegid32","geteuid","geteuid32","getgid","getgid32","getgroups","getgroups32","getitimer","get_mempolicy","getpeername","getpgid","getpgrp","getpid","getppid","getpriority","getrandom","getresgid","getresgid32","getresuid","getresuid32","getrlimit","getrusage","getsid","getsockname","getsockopt","gettid","gettimeofday","getuid","getuid32","getxattr","inotify_add_watch","inotify_init","inotify_init1","inotify_rm_watch","io_cancel","io_destroy","io_getevents","io_setup","io_submit","ioctl","ioprio_get","ioprio_set","ipc","keyctl","kill","lchown","lchown32","lgetxattr","link","linkat","listen","listxattr","llistxattr","lremovexattr","lseek","lsetxattr","lstat","lstat64","madvise","mbind","memfd_create","memfd_secret","mincore","mkdir","mkdirat","mknod","mknodat","mlock","mlock2","mlockall","mmap","mmap2","mount","move_mount","mprotect","mq_getsetattr","mq_notify","mq_open","mq_timedreceive","mq_timedreceive_time64","mq_timedsend","mq_timedsend_time64","mq_unlink","mremap","msgctl","msgget","msgrcv","msgsnd","msync","munlock","munlockall","munmap","name_to_handle_at","nanosleep","newfstatat","open","openat","openat2","open_tree","pause","pidfd_getfd","pidfd_open","pidfd_send_signal","pipe","pipe2","pivot_root","pkey_alloc","pkey_free","pkey_mprotect","poll","ppoll","ppoll_time64","prctl","pread64","preadv","preadv2","prlimit64","pselect6","pselect6_time64","pwrite64","pwritev","pwritev2","read","readahead","readdir","readlink","readlinkat","readv","reboot","recv","recvfrom","recvmmsg","recvmmsg_time64","recvmsg","remap_file_pages","removexattr","rename","renameat","renameat2","restart_syscall","rmdir","rseq","rt_sigaction","rt_sigpending","rt_sigprocmask","rt_sigqueueinfo","rt_sigreturn","rt_sigsuspend","rt_sigtimedwait","rt_sigtimedwait_time64","rt_tgsigqueueinfo","sched_get_priority_max","sched_get_priority_min","sched_getaffinity","sched_getattr","sched_getparam","sched_getscheduler","sched_rr_get_interval","sched_rr_get_interval_time64","sched_setaffinity","sched_setattr","sched_setparam","sched_setscheduler","sched_yield","seccomp","select","semctl","semget","semop","semtimedop","semtimedop_time64","send","sendfile","sendfile64","sendmmsg","sendmsg","sendto","setns","set_mempolicy","set_robust_list","set_thread_area","set_tid_address","setfsgid","setfsgid32","setfsuid","setfsuid32","setgid","setgid32","setgroups","setgroups32","setitimer","setpgid","setpriority","setregid","setregid32","setresgid","setresgid32","setresuid","setresuid32","setreuid","setreuid32","setrlimit","setsid","setsockopt","setuid","setuid32","setxattr","shmat","shmctl","shmdt","shmget","shutdown","sigaltstack","signalfd","signalfd4","sigreturn","socketcall","socketpair","splice","stat","stat64","statfs","statfs64","statx","symlink","symlinkat","sync","sync_file_range","syncfs","sysinfo","syslog","tee","tgkill","time","timer_create","timer_delete","timer_getoverrun","timer_gettime","timer_gettime64","timer_settime","timer_settime64","timerfd_create","timerfd_gettime","timerfd_gettime64","timerfd_settime","timerfd_settime64","times","tkill","truncate","truncate64","ugetrlimit","umask","umount","umount2","uname","unlink","unlinkat","unshare","utime","utimensat","utimensat_time64","utimes","vfork","wait4","waitid","waitpid","write","writev"],"action":"SCMP_ACT_ALLOW"},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":0,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":8,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":131072,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":131080,"op":"SCMP_CMP_EQ"}]},{"names":["personality"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":4294967295,"op":"SCMP_CMP_EQ"}]},{"names":["arm_fadvise64_64","arm_sync_file_range","sync_file_range2","breakpoint","cacheflush","set_tls"],"action":"SCMP_ACT_ALLOW"},{"names":["open_by_handle_at"],"action":"SCMP_ACT_ERRNO","errnoRet":1},{"names":["bpf","fanotify_init","lookup_dcookie","perf_event_open","quotactl","setdomainname","sethostname","setns"],"action":"SCMP_ACT_ERRNO","errnoRet":1},{"names":["chroot"],"action":"SCMP_ACT_ALLOW"},{"names":["delete_module","init_module","finit_module","query_module"],"action":"SCMP_ACT_ERRNO","errnoRet":1},{"names":["acct"],"action":"SCMP_ACT_ERRNO","errnoRet":1},{"names":["kcmp","process_madvise","process_vm_readv","process_vm_writev","ptrace"],"action":"SCMP_ACT_ERRNO","errnoRet":1},{"names":["iopl","ioperm"],"action":"SCMP_ACT_ERRNO","errnoRet":1},{"names":["settimeofday","stime","clock_settime","clock_settime64"],"action":"SCMP_ACT_ERRNO","errnoRet":1},{"names":["vhangup"],"action":"SCMP_ACT_ERRNO","errnoRet":1},{"names":["socket"],"action":"SCMP_ACT_ERRNO","errnoRet":22,"args":[{"index":0,"value":16,"op":"SCMP_CMP_EQ"},{"index":2,"value":9,"op":"SCMP_CMP_EQ"}]},{"names":["socket"],"action":"SCMP_ACT_ALLOW","args":[{"index":2,"value":9,"op":"SCMP_CMP_NE"}]},{"names":["socket"],"action":"SCMP_ACT_ALLOW","args":[{"index":0,"value":16,"op":"SCMP_CMP_NE"}]},{"names":["socket"],"action":"SCMP_ACT_ALLOW","args":[{"index":2,"value":9,"op":"SCMP_CMP_NE"}]}]},"maskedPaths":["/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware","/sys/fs/selinux","/sys/dev","/sys/fs/cgroup"],"readonlyPaths":["/proc/asound","/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]}} 
DEBU[0000] Running ["/usr/sbin/runc" "--systemd-cgroup" "create" "--bundle" "/var/tmp/buildah1094495047" "--pid-file" "/var/tmp/buildah1094495047/pid" "--no-new-keyring" "buildah-buildah1094495047"] 
DEBU[0000] "/var/tmp/buildah1094495047/mnt/rootfs" is apparently not really mounted, skipping 
DEBU[0000] "/var/tmp/buildah1094495047/mnt" is apparently not really mounted, skipping 
error running container: error from /usr/sbin/runc creating container for [/bin/sh -c date]: time="2022-11-17T17:40:47Z" level=warning msg="unable to get oom kill count" error="openat2 /sys/fs/cgroup/system.slice/runc-buildah-buildah1094495047.scope/memory.events: no such file or directory"
time="2022-11-17T17:40:47Z" level=error msg="runc create failed: unable to start container process: unable to apply cgroup configuration: unable to start unit \"runc-buildah-buildah1094495047.scope\" (properties [{Name:Description Value:\"libcontainer container buildah-buildah1094495047\"} {Name:Slice Value:\"system.slice\"} {Name:Delegate Value:true} {Name:PIDs Value:@au [7555]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Interactive authentication required."
: exit status 1
DEBU[0004] error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[date] Flags:[] Attrs:map[] Message:RUN date Original:RUN date}: error while running runtime: exit status 1 
Error: error building at STEP "RUN date": error while running runtime: exit status 1
CircleCI received exit code 0

Does it indicate something useful?

giuseppe commented 2 years ago

runc is failing for the same reason.

sudo doesn't create a user session, could you try with machinectl shell $USER@?

giuseppe commented 2 years ago

alternatively, you could just use --cgroup-manager cgroupfs and you'll avoid using systemd to create the cgroup

adelton commented 2 years ago

While sudo might not create a user session, that sudo runs

sudo -niHu circleci -- /tmp/circleci-machine-agent3174475418/circleci-agent _internal agent-runner --config -

which I assume might do something about the session.

After all, loginctl session-status shows

c1 - circleci (1001)
       Since: Sat 2022-11-19 06:25:37 UTC; 46s ago
      Leader: 906 (sudo)
      Remote: user root
     Service: sudo-i; type unspecified; class background
       State: active
        Unit: session-c1.scope
          ├─ 906 sudo -niHu circleci -- /tmp/circleci-machine-agent3174475418/circleci-agent _internal agent-runner --config -
          ├─ 915 /tmp/circleci-machine-agent3174475418/circleci-agent _internal agent-runner --config -
          ├─3745 podman
          ├─4786 /bin/bash -eo pipefail -c "loginctl session-status | cat"
          ├─5109 loginctl session-status
          └─5110 cat

showing a session for uid 1001, not for root.

adelton commented 2 years ago

Using --cgroup-manager cgroupfs worked, thanks for that pointer: https://app.circleci.com/pipelines/github/adelton/freeipa-container/113/workflows/d0c21c49-b7a1-4020-910a-d48a7bbcffc9/jobs/519

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 1 year ago

I don't believe this is still and issue and you have a workaround closing.

carlosrodfern commented 1 year ago

@rhatdan , I'm running into this same issue in Fedora 38. I'm logged in as a normal user, and when I try to build something simple, it shows the same problem. This was not a problem before. XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS are both set.

$ cat Containerfile 
FROM quay.io/centos/centos:stream9

RUN dnf install -y bind-utils
podman build -t test:latest .
STEP 1/2: FROM quay.io/centos/centos:stream9
STEP 2/2: RUN dnf install -y bind-utils
error running container: from /usr/bin/crun creating container for [/bin/sh -c dnf install -y bind-utils]: sd-bus call: Interactive authentication required.: Permission denied
: exit status 1
ERRO[0000] did not get container create message from subprocess: EOF 
Error: building at STEP "RUN dnf install -y bind-utils": while running runtime: exit status 1

When I use the workaround, it works: podman --cgroup-manager cgroupfs build --log-level=warning -t test:latest ., however, having to add that flag every time for a desktop user that it is logged in, as their first experience, rather shows there is a bug somewhere when using systemd as the cgroup manager.

Is this a podman issue or a systemd issue or some misconfiguration in f38? Is there anything I should look into?

Thank you.

rhatdan commented 1 year ago

Please open a new issue, unless you are running under CircleCI?

carlosrodfern commented 1 year ago

Will do. Thank you @rhatdan