search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.72k stars 2.41k forks source link

Error: retrieving exec session ... exit code: exec died event for session ... (container ...) not found: unable to find event #20018

Open PlatonB opened 1 year ago

PlatonB commented 1 year ago

Issue Description

GNOME OS Development (45) podman version 4.6.2

Steps to reproduce the issue

podman run --name mongodb -d --restart always -p 27017:27017 -v /home/platon/MongoDB:/data/db mongodb/mongodb-community-server
Resolving "mongodb/mongodb-community-server" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/mongodb/mongodb-community-server:latest...
Getting image source signatures
Copying blob 20c07e475edb done  
Copying blob 77c840be1645 done  
Copying blob a07f81f5a82b done  
Copying blob 89986b8531f9 done  
Copying blob 89b3eb101e94 done  
Copying blob 445a6a12be2b done  
Copying blob 962abecc5db9 done  
Copying blob 298898982669 done  
Copying blob 4f4fb700ef54 done  
Copying blob ab2098a9dcb6 done  
Copying config 283ff3a640 done  
Writing manifest to image destination
89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7
podman exec -it mongodb mongosh
Error: retrieving exec session 4831a80c1992e93b4d2920a6b259e4dfd18eb05bafcc07df50bd649db75666a3 exit code: exec died event for session 4831a80c1992e93b4d2920a6b259e4dfd18eb05bafcc07df50bd649db75666a3 (container 89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7) not found: unable to find event

Describe the results you received

See "Steps to reproduce the issue"

Describe the results you expected

Normally, the podman exec -it mongodb mongosh command should open the MongoDB shell.

podman info output

podman info
host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d-dirty'
  cpuUtilization:
    idlePercent: 59.97
    systemPercent: 6.01
    userPercent: 34.02
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: org.gnome.gnomeos
    variant: devel
    version: "45"
  eventLogger: journald
  freeLocks: 2034
  hostname: 10.0.2.15
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.5.0
  linkmode: dynamic
  logDriver: journald
  memFree: 137895936
  memTotal: 8311472128
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 1.8.7-dirty
      commit: 53a9996ce82d1ee818349bdcc64797a1fa0433c4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: /usr/local/bin/pasta
    package: Unknown
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.1
      commit: unknown
      libslirp: 4.7.0-dirty
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 0h 31m 32.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/platon/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/platon/.local/share/containers/storage
  graphRootAllocated: 105101250560
  graphRootUsed: 31648763904
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/platon/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1320937200
  BuiltTime: Thu Nov 10 19:00:00 2011
  GitCommit: 5db42e86862ef42c59304c38aa583732fd80f178-dirty
  GoVersion: go1.21.0
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Gnome Boxes (flatpak)

Additional information

podman inspect mongodb
[
     {
          "Id": "89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7",
          "Created": "2023-09-18T18:33:05.962561853+03:00",
          "Path": "python3",
          "Args": [
               "/usr/local/bin/docker-entrypoint.py",
               "mongod"
          ],
          "State": {
               "OciVersion": "1.1.0-rc.3",
               "Status": "running",
               "Running": true,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 159222,
               "ConmonPid": 159220,
               "ExitCode": 0,
               "Error": "",
               "StartedAt": "2023-09-18T18:50:01.312035915+03:00",
               "FinishedAt": "2023-09-18T18:50:01.127846696+03:00",
               "Health": {
                    "Status": "",
                    "FailingStreak": 0,
                    "Log": null
               },
               "CgroupPath": "/user.slice/user-1000.slice/user@1000.service/user.slice/libpod-89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7.scope",
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          },
          "Image": "283ff3a640d8c31295bef50bf1cda739dd66cde6bc362c10e3d8f4775dff5c17",
          "ImageDigest": "sha256:e3b7015c2888611ccc6faaaf9d995108b7cb69ee5b0c62f09a70d07733e8ca7d",
          "ImageName": "docker.io/mongodb/mongodb-community-server:latest",
          "Rootfs": "",
          "Pod": "",
          "ResolvConfPath": "/run/user/1000/containers/vfs-containers/89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7/userdata/resolv.conf",
          "HostnamePath": "/run/user/1000/containers/vfs-containers/89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7/userdata/hostname",
          "HostsPath": "/run/user/1000/containers/vfs-containers/89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7/userdata/hosts",
          "StaticDir": "/home/platon/.local/share/containers/storage/vfs-containers/89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7/userdata",
          "OCIConfigPath": "/home/platon/.local/share/containers/storage/vfs-containers/89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/user/1000/containers/vfs-containers/89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7/userdata/conmon.pid",
          "PidFile": "/run/user/1000/containers/vfs-containers/89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7/userdata/pidfile",
          "Name": "mongodb",
          "RestartCount": 3308,
          "Driver": "vfs",
          "MountLabel": "",
          "ProcessLabel": "",
          "AppArmorProfile": "",
          "EffectiveCaps": null,
          "BoundingCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "ExecIDs": [],
          "GraphDriver": {
               "Name": "vfs",
               "Data": null
          },
          "Mounts": [
               {
                    "Type": "volume",
                    "Name": "166018087ab6d95cb7e07fe17629033aa7f466253f7ad2c7cc2b5edfb3fc887c",
                    "Source": "/home/platon/.local/share/containers/storage/volumes/166018087ab6d95cb7e07fe17629033aa7f466253f7ad2c7cc2b5edfb3fc887c/_data",
                    "Destination": "/data/configdb",
                    "Driver": "local",
                    "Mode": "",
                    "Options": [
                         "nodev",
                         "exec",
                         "nosuid",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "bind",
                    "Source": "/home/platon/MongoDB",
                    "Destination": "/data/db",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               }
          ],
          "Dependencies": [],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "",
               "IPAddress": "",
               "IPPrefixLen": 0,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {
                    "27017/tcp": [
                         {
                              "HostIp": "",
                              "HostPort": "27017"
                         }
                    ]
               },
               "SandboxKey": "/run/user/1000/netns/netns-4d0eead3-db74-747f-3c7d-7f7dfdd0f56d"
          },
          "Namespace": "",
          "IsInfra": false,
          "IsService": false,
          "KubeExitCodePropagation": "invalid",
          "lockNumber": 0,
          "Config": {
               "Hostname": "89e2ce9aa76c",
               "Domainname": "",
               "User": "mongodb",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": false,
               "OpenStdin": false,
               "StdinOnce": false,
               "Env": [
                    "container=podman",
                    "HOME=/data/db",
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "TERM=xterm",
                    "HOSTNAME=89e2ce9aa76c"
               ],
               "Cmd": [
                    "mongod"
               ],
               "Image": "docker.io/mongodb/mongodb-community-server:latest",
               "Volumes": null,
               "WorkingDir": "/",
               "Entrypoint": "python3 /usr/local/bin/docker-entrypoint.py",
               "OnBuild": null,
               "Labels": {
                    "description": "Container configured with a standalone instance of MongoDB",
                    "maintainer": "support@mongodb.com",
                    "name": "MongoDB Standalone",
                    "org.opencontainers.image.ref.name": "ubuntu",
                    "org.opencontainers.image.version": "22.04",
                    "summary": "MongoDB Standalone Container",
                    "vendor": "MongoDB",
                    "version": "7.0.1"
               },
               "Annotations": {
                    "io.container.manager": "libpod",
                    "org.opencontainers.image.stopSignal": "15"
               },
               "StopSignal": 15,
               "HealthcheckOnFailureAction": "none",
               "CreateCommand": [
                    "podman",
                    "run",
                    "--name",
                    "mongodb",
                    "-d",
                    "--restart",
                    "always",
                    "-p",
                    "27017:27017",
                    "-v",
                    "/home/platon/MongoDB:/data/db",
                    "mongodb/mongodb-community-server"
               ],
               "Umask": "0022",
               "Timeout": 0,
               "StopTimeout": 10,
               "Passwd": true,
               "sdNotifyMode": "container"
          },
          "HostConfig": {
               "Binds": [
                    "166018087ab6d95cb7e07fe17629033aa7f466253f7ad2c7cc2b5edfb3fc887c:/data/configdb:rprivate,rw,nodev,exec,nosuid,rbind",
                    "/home/platon/MongoDB:/data/db:rw,rprivate,nodev,rbind"
               ],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "0B"
               },
               "NetworkMode": "slirp4netns",
               "PortBindings": {
                    "27017/tcp": [
                         {
                              "HostIp": "",
                              "HostPort": "27017"
                         }
                    ]
               },
               "RestartPolicy": {
                    "Name": "always",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [],
               "CapDrop": [],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [],
               "GroupAdd": [],
               "IpcMode": "shareable",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": false,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [],
               "Tmpfs": {},
               "UTSMode": "private",
               "UsernsMode": "",
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 0,
               "NanoCpus": 0,
               "CgroupParent": "user.slice",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 0,
               "MemorySwappiness": 0,
               "OomKillDisable": false,
               "PidsLimit": 2048,
               "Ulimits": [
                    {
                         "Name": "RLIMIT_NOFILE",
                         "Soft": 524288,
                         "Hard": 524288
                    },
                    {
                         "Name": "RLIMIT_NPROC",
                         "Soft": 31513,
                         "Hard": 31513
                    }
               ],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          }
     }
]
mheon commented 1 year ago

Can the user running Podman access the journal (simple check is to run journalctl; if it doesn't error and you get log output, then the user has access).

PlatonB commented 1 year ago 
journalctl
сен 02 20:02:29 localhost kernel: Linux version 5.19.17 (tomjon@buildstream) (gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.38) #1 SMP PREEMPT_DYNAMIC Thu Nov 10 15:00:00 UTC 2011
сен 02 20:02:29 localhost kernel: Command line: initrd=\ostree\gnome-os-a48ee442a9a85236aafd59af80ceb1b50fe9fd474c08c8f2b2d55c016d3f1d41\initramfs-5.19.17.img rw quiet splash console=tty0 ostree=/ostree/boot.0/gnome-os/a48ee442a9a85236aa>
...
mheon commented 1 year ago

Two things:

  1. Can you run podman events in a separate terminal before running podman exec and see what events appear?
  2. Can you check the journal after running podman exec and see if Podman added any logs?
PlatonB commented 1 year ago

podman events endlessly repeats this strings output:

2023-09-20 10:15:44.651305503 +0300 MSK container init 89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7 (image=docker.io/mongodb/mongodb-community-server:latest, name=mongodb, maintainer=support@mongodb.com, name=MongoDB Standalone, org.opencontainers.image.ref.name=ubuntu, org.opencontainers.image.version=22.04, summary=MongoDB Standalone Container, vendor=MongoDB, version=7.0.1, description=Container configured with a standalone instance of MongoDB)
2023-09-20 10:15:44.658146379 +0300 MSK container start 89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7 (image=docker.io/mongodb/mongodb-community-server:latest, name=mongodb, summary=MongoDB Standalone Container, vendor=MongoDB, version=7.0.1, description=Container configured with a standalone instance of MongoDB, maintainer=support@mongodb.com, name=MongoDB Standalone, org.opencontainers.image.ref.name=ubuntu, org.opencontainers.image.version=22.04)
2023-09-20 10:15:44.830002278 +0300 MSK container died 89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7 (image=docker.io/mongodb/mongodb-community-server:latest, name=mongodb, name=MongoDB Standalone, org.opencontainers.image.ref.name=ubuntu, org.opencontainers.image.version=22.04, summary=MongoDB Standalone Container, vendor=MongoDB, version=7.0.1, description=Container configured with a standalone instance of MongoDB, maintainer=support@mongodb.com)
2023-09-20 10:15:44.848983499 +0300 MSK container restart 89e2ce9aa76c5f822adc1fa66c01198281a3cc037518b20c33aa0277466aa1d7 (image=docker.io/mongodb/mongodb-community-server:latest, name=mongodb, maintainer=support@mongodb.com, name=MongoDB Standalone, org.opencontainers.image.ref.name=ubuntu, org.opencontainers.image.version=22.04, summary=MongoDB Standalone Container, vendor=MongoDB, version=7.0.1, description=Container configured with a standalone instance of MongoDB)

Attempting to run podman exec:

podman exec
Error: exec requires the name or ID of a container or the --latest flag
podman exec mongodb
Error: must provide a non-empty command to start an exec session: invalid argument
podman exec 89e2ce9aa76c
Error: must provide a non-empty command to start an exec session: invalid argument
podman exec --latest
Error: must provide a non-empty command to start an exec session: invalid argument
github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 1 year ago

@mheon @PlatonB Is this still an issue?

mheon commented 1 year ago

The event is definitely there (we can see it in podman events) so it's very interesting that podman exec can't find it. Would need to be able to reproduce to debug much further, and I can't make this happen on my system. Possibly a journal configuration issue?

mheon commented 1 year ago

Though - that podman events trace is only container events, not exec events, so a trace after a successful podman exec into the container would be more enlightening.

PlatonB commented 1 year ago

@mheon @PlatonB Is this still an issue?

@rhatdan Yes(. The Podman version is more recent (4.7.1), but Mongodb Shell still doesn't run.

Would need to be able to reproduce to debug much further, and I can't make this happen on my system.

@mheon Have you tried it specifically on the devel branch of Gnome OS 45?