search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.64k stars 2.41k forks source link

[Bug]: Unable to hijack connection with trace log enabled in server mode #17332

Closed Agalin closed 1 year ago

Agalin commented 1 year ago

Issue Description

When you run Podman server (local, through unix socket) with log level set to trace it fails to start a container. Podman client ends with Error: unable to hijack connection.

When you run the same container with same options directly (without using remote) it starts properly with trace log level.

Steps to reproduce the issue

Steps to reproduce the issue

  1. Start podman server with --log-level=trace (podman --log-level=trace system service)
  2. Configure podman client to use that server through CONTAINER_HOST env variable.
  3. Try podman run --rm -ti fedora.

Describe the results you received

podman run command fails with Error: unable to hijack connection.

In server logs it seems to have problem with container runtime initialization:

time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument"
time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument"
time="2023-02-02T12:38:05Z" level=trace msg="found runtime \"/usr/bin/crun\""
time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument"
time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument"
time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument"
time="2023-02-02T12:38:05Z" level=debug msg="Using OCI runtime \"/usr/bin/crun\""
time="2023-02-02T12:38:05Z" level=info msg="Setting parallel job count to 13"
time="2023-02-02T12:38:05Z" level=debug msg="Error looking up container \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\": no container with name or ID \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\" found: no such container"
time="2023-02-02T12:38:05Z" level=error msg="Running container cleanup: no container with name or ID \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\" found: no such container"
Full log ``` time="2023-02-02T12:38:05Z" level=info msg="/usr/bin/podman filtering at log level trace" time="2023-02-02T12:38:05Z" level=debug msg="Called service.PersistentPreRunE(/usr/bin/podman --log-level=trace system service)" time="2023-02-02T12:38:05Z" level=trace msg="Reading configuration file \"/usr/share/containers/containers.conf\"" time="2023-02-02T12:38:05Z" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\"" time="2023-02-02T12:38:05Z" level=trace msg="&{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.50.1 Annotations:[] BaseHostsFile: CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HostContainersInternalIP: HTTPProxy:true Init:false InitPath: IPCNS:shareable LogDriver:journald LogSizeMax:-1 LogTag: NetNS:private NoHosts:false PidsLimit:2048 PidNS:private PrepareVolumeOnCreate:false SeccompProfile: ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS: UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ConmonRsPath:[/usr/libexec/podman/conmonrs /usr/local/libexec/podman/conmonrs /usr/local/lib/podman/conmonrs /usr/bin/conmonrs /usr/sbin/conmonrs /usr/local/bin/conmonrs /usr/local/sbin/conmonrs /run/current-system/sw/bin/conmonrs] CompatAPIEnforceDockerHub:true DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath: EventsLogFileMaxSize:1000000 EventsLogger:journald graphRoot:/home/gitlab-runner/.local/share/containers/storage HelperBinariesDir:[/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: ImageVolumeMode:bind InfraCommand: InfraImage: InitPath:/usr/libexec/podman/catatonit LockType:shm MachineEnabled:false MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] krun:[/usr/bin/krun /usr/local/bin/krun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] runj:[/usr/local/bin/runj] runsc:[/usr/bin/runsc /usr/sbin/runsc /usr/local/bin/runsc /usr/local/sbin/runsc /bin/runsc /sbin/runsc /run/current-system/sw/bin/runsc]] PodExitPolicy:continue PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] SSHConfig:/home/gitlab-runner/.ssh/config RuntimePath:[] RuntimeSupportsJSON:[crun runc kata runsc krun] RuntimeSupportsNoCgroups:[crun krun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc krun] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 ServiceTimeout:5 StaticDir:/home/gitlab-runner/.local/share/containers/storage/libpod StopTimeout:10 ExitCommandDelay:300 ImageCopyTmpDir:/var/tmp TmpDir:/run/user/993/libpod/tmp VolumePath:/home/gitlab-runner/.local/share/containers/storage/volumes VolumePluginTimeout:5 VolumePlugins:map[] ChownCopiedFiles:true CompressionFormat:} Machine:{CPUs:1 DiskSize:100 Image:testing Memory:2048 User:core Volumes:[$HOME:$HOME]} Network:{NetworkBackend: CNIPluginDirs:[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] DefaultNetwork:podman DefaultSubnet:10.88.0.0/16 DefaultSubnetPools:[{Base:172.18.0.64/26 Size:29} {Base:172.18.0.128/25 Size:29} {Base:10.92.0.0/14 Size:24} {Base:10.96.0.0/11 Size:24} {Base:10.128.0.0/9 Size:24}] NetworkConfigDir: DNSBindPort:0} Secrets:{Driver:file Opts:map[]} ConfigMaps:{Driver: Opts:map[]}}" time="2023-02-02T12:38:05Z" level=trace msg="Reading configuration file \"/etc/containers/containers.conf\"" time="2023-02-02T12:38:05Z" level=debug msg="Failed to decode the keys [\"storage\" \"storage.driver\"] from \"/etc/containers/containers.conf\"." time="2023-02-02T12:38:05Z" level=debug msg="Merged system config \"/etc/containers/containers.conf\"" time="2023-02-02T12:38:05Z" level=trace msg="&{Containers:{Devices:[] Volumes:[/etc/ssl/:/etc/ssl/:O /etc/pki/ca-trust/extracted/pem/:/etc/pki/ca-trust/extracted/pem/:O /etc/pki/tls/:/etc/pki/tls/:O] ApparmorProfile:containers-default-0.50.1 Annotations:[] BaseHostsFile: CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[10.0.2.3] DNSOptions:[edns0] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HostContainersInternalIP: HTTPProxy:true Init:false InitPath: IPCNS:shareable LogDriver:journald LogSizeMax:-1 LogTag: NetNS:private NoHosts:false PidsLimit:2048 PidNS:private PrepareVolumeOnCreate:false SeccompProfile: ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS: UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ConmonRsPath:[/usr/libexec/podman/conmonrs /usr/local/libexec/podman/conmonrs /usr/local/lib/podman/conmonrs /usr/bin/conmonrs /usr/sbin/conmonrs /usr/local/bin/conmonrs /usr/local/sbin/conmonrs /run/current-system/sw/bin/conmonrs] CompatAPIEnforceDockerHub:true DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath: EventsLogFileMaxSize:1000000 EventsLogger:journald graphRoot:/home/gitlab-runner/.local/share/containers/storage HelperBinariesDir:[/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: ImageVolumeMode:bind InfraCommand: InfraImage: InitPath:/usr/libexec/podman/catatonit LockType:shm MachineEnabled:false MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[mtu=48000 enable_ipv6=false] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] krun:[/usr/bin/krun /usr/local/bin/krun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] runj:[/usr/local/bin/runj] runsc:[/usr/bin/runsc /usr/sbin/runsc /usr/local/bin/runsc /usr/local/sbin/runsc /bin/runsc /sbin/runsc /run/current-system/sw/bin/runsc]] PodExitPolicy:continue PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] SSHConfig:/home/gitlab-runner/.ssh/config RuntimePath:[] RuntimeSupportsJSON:[crun runc kata runsc krun] RuntimeSupportsNoCgroups:[crun krun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc krun] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 ServiceTimeout:5 StaticDir:/home/gitlab-runner/.local/share/containers/storage/libpod StopTimeout:10 ExitCommandDelay:300 ImageCopyTmpDir:/var/tmp TmpDir:/run/user/993/libpod/tmp VolumePath:/home/gitlab-runner/.local/share/containers/storage/volumes VolumePluginTimeout:5 VolumePlugins:map[] ChownCopiedFiles:true CompressionFormat:} Machine:{CPUs:1 DiskSize:100 Image:testing Memory:2048 User:core Volumes:[$HOME:$HOME]} Network:{NetworkBackend: CNIPluginDirs:[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] DefaultNetwork:podman DefaultSubnet:172.18.0.1/26 DefaultSubnetPools:[{Base:172.18.0.64/26 Size:29} {Base:172.18.0.128/25 Size:29}] NetworkConfigDir: DNSBindPort:0} Secrets:{Driver:file Opts:map[]} ConfigMaps:{Driver: Opts:map[]}}" time="2023-02-02T12:38:05Z" level=debug msg="Using conmon: \"/usr/bin/conmon\"" time="2023-02-02T12:38:05Z" level=debug msg="Initializing boltdb state at /home/gitlab-runner/.local/share/containers/storage/libpod/bolt_state.db" time="2023-02-02T12:38:05Z" level=debug msg="systemd-logind: Unknown object '/'." time="2023-02-02T12:38:05Z" level=debug msg="Using graph driver overlay" time="2023-02-02T12:38:05Z" level=debug msg="Using graph root /home/gitlab-runner/.local/share/containers/storage" time="2023-02-02T12:38:05Z" level=debug msg="Using run root /run/user/993/containers" time="2023-02-02T12:38:05Z" level=debug msg="Using static dir /home/gitlab-runner/.local/share/containers/storage/libpod" time="2023-02-02T12:38:05Z" level=debug msg="Using tmp dir /run/user/993/libpod/tmp" time="2023-02-02T12:38:05Z" level=debug msg="Using volume path /home/gitlab-runner/.local/share/containers/storage/volumes" time="2023-02-02T12:38:05Z" level=debug msg="Set libpod namespace to \"\"" time="2023-02-02T12:38:05Z" level=debug msg="[graphdriver] trying provided driver \"overlay\"" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that overlay is supported" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that overlay is supported" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that metacopy is not being used" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that native-diff is usable" time="2023-02-02T12:38:05Z" level=debug msg="backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false" time="2023-02-02T12:38:05Z" level=debug msg="Initializing event backend journald" time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument" time="2023-02-02T12:38:05Z" level=trace msg="found runtime \"/usr/bin/crun\"" time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument" time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument" time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument" time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument" time="2023-02-02T12:38:05Z" level=debug msg="Using OCI runtime \"/usr/bin/crun\"" time="2023-02-02T12:38:05Z" level=info msg="Setting parallel job count to 13" time="2023-02-02T12:38:05Z" level=info msg="Using systemd socket activation to determine API endpoint" time="2023-02-02T12:38:05Z" level=debug msg="registered SIGHUP watcher for config" time="2023-02-02T12:38:05Z" level=info msg="API service listening on \"/run/user/993/podman/podman.sock\". URI: \"/run/user/993/podman/podman.sock\"" time="2023-02-02T12:38:05Z" level=debug msg="CORS Headers were not set" time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:new 0m+0h/0t connection(s)" X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:active 0m+0h/1t connection(s)" X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=trace msg=OK API=response X-Reference-Id=0xc00056e008 @ - - [02/Feb/2023:12:38:05 +0000] "GET /v4.3.1/libpod/_ping HTTP/1.1" 200 2 "" "Go-http-client/1.1" time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:idle 1m+0h/1t connection(s)" X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:closed 1m+0h/1t connection(s)" X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:new 0m+0h/1t connection(s)" X-Reference-Id=0xc000360008 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:active 0m+0h/2t connection(s)" X-Reference-Id=0xc000360008 time="2023-02-02T12:38:05Z" level=trace msg=OK API=response X-Reference-Id=0xc000360008 @ - - [02/Feb/2023:12:38:05 +0000] "GET /v4.3.1/libpod/_ping HTTP/1.1" 200 2 "" "Go-http-client/1.1" time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:idle 1m+0h/2t connection(s)" X-Reference-Id=0xc000360008 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:closed 1m+0h/2t connection(s)" X-Reference-Id=0xc000360008 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:new 0m+0h/2t connection(s)" X-Reference-Id=0xc00056e050 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:active 0m+0h/3t connection(s)" X-Reference-Id=0xc00056e050 time="2023-02-02T12:38:05Z" level=debug msg="Pulling image fedora (policy: missing)" time="2023-02-02T12:38:05Z" level=debug msg="Looking up image \"fedora\" in local containers storage" time="2023-02-02T12:38:05Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux [] }" time="2023-02-02T12:38:05Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\"" time="2023-02-02T12:38:05Z" level=debug msg="Failed to decode keys [\"registry.mirror.prefix\" \"registry.mirror.prefix\" \"registry.mirror.prefix\" \"registry.mirror.prefix\" \"registry.mirror.prefix\" \"registry.mirror.prefix\" \"registry.mirror.prefix\"] from \"/etc/containers/registries.conf\"" time="2023-02-02T12:38:05Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\"" time="2023-02-02T12:38:05Z" level=debug msg="Trying \"registry.fedoraproject.org/fedora:latest\" ..." time="2023-02-02T12:38:05Z" level=debug msg="parsed reference into \"[overlay@/home/gitlab-runner/.local/share/containers/storage+/run/user/993/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=debug msg="Found image \"fedora\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage" time="2023-02-02T12:38:05Z" level=debug msg="Found image \"fedora\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage ([overlay@/home/gitlab-runner/.local/share/containers/storage+/run/user/993/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)" time="2023-02-02T12:38:05Z" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=debug msg="Looking up image \"registry.fedoraproject.org/fedora:latest\" in local containers storage" time="2023-02-02T12:38:05Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux [] }" time="2023-02-02T12:38:05Z" level=debug msg="Trying \"registry.fedoraproject.org/fedora:latest\" ..." time="2023-02-02T12:38:05Z" level=debug msg="parsed reference into \"[overlay@/home/gitlab-runner/.local/share/containers/storage+/run/user/993/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=debug msg="Found image \"registry.fedoraproject.org/fedora:latest\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage" time="2023-02-02T12:38:05Z" level=debug msg="Found image \"registry.fedoraproject.org/fedora:latest\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage ([overlay@/home/gitlab-runner/.local/share/containers/storage+/run/user/993/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)" time="2023-02-02T12:38:05Z" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=trace msg="{\"images\":[\"19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"],\"id\":\"19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"}\n" API=response X-Reference-Id=0xc00056e050 @ - - [02/Feb/2023:12:38:05 +0000] "POST /v4.3.1/libpod/images/pull?alltags=false&arch=&authfile=&os=&password=&policy=missing&quiet=false&reference=fedora&username=&variant= HTTP/1.1" 200 152 "" "Go-http-client/1.1" time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:idle 1m+0h/3t connection(s)" X-Reference-Id=0xc00056e050 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:new 1m+0h/3t connection(s)" X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:active 1m+0h/4t connection(s)" X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=debug msg="Looking up image \"fedora\" in local containers storage" time="2023-02-02T12:38:05Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux [] }" time="2023-02-02T12:38:05Z" level=debug msg="Trying \"registry.fedoraproject.org/fedora:latest\" ..." time="2023-02-02T12:38:05Z" level=debug msg="parsed reference into \"[overlay@/home/gitlab-runner/.local/share/containers/storage+/run/user/993/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=debug msg="Found image \"fedora\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage" time="2023-02-02T12:38:05Z" level=debug msg="Found image \"fedora\" as \"registry.fedoraproject.org/fedora:latest\" in local containers storage ([overlay@/home/gitlab-runner/.local/share/containers/storage+/run/user/993/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)" time="2023-02-02T12:38:05Z" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=debug msg="Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a" time="2023-02-02T12:38:05Z" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=debug msg="Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a" time="2023-02-02T12:38:05Z" level=debug msg="Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a" time="2023-02-02T12:38:05Z" level=debug msg="Inspecting image 19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a" time="2023-02-02T12:38:05Z" level=debug msg="User mount /etc/ssl/:/etc/ssl/ options [O]" time="2023-02-02T12:38:05Z" level=debug msg="User mount /etc/pki/ca-trust/extracted/pem/:/etc/pki/ca-trust/extracted/pem/ options [O]" time="2023-02-02T12:38:05Z" level=debug msg="User mount /etc/pki/tls/:/etc/pki/tls/ options [O]" time="2023-02-02T12:38:05Z" level=debug msg="using systemd mode: false" time="2023-02-02T12:38:05Z" level=debug msg="No hostname set; container's hostname will default to runtime default" time="2023-02-02T12:38:05Z" level=debug msg="Loading seccomp profile from \"/usr/share/containers/seccomp.json\"" time="2023-02-02T12:38:05Z" level=debug msg="Allocated lock 8 for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=debug msg="parsed reference into \"[overlay@/home/gitlab-runner/.local/share/containers/storage+/run/user/993/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=debug msg="exporting opaque data as blob \"sha256:19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that idmapped mounts for overlay are not supported" time="2023-02-02T12:38:05Z" level=debug msg="Check for idmapped mounts support " time="2023-02-02T12:38:05Z" level=debug msg="Created container \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\"" time="2023-02-02T12:38:05Z" level=debug msg="Container \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\" has work directory \"/home/gitlab-runner/.local/share/containers/storage/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata\"" time="2023-02-02T12:38:05Z" level=debug msg="Container \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\" has run directory \"/run/user/993/containers/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata\"" 2023-02-02 12:38:05.18881356 +0000 UTC m=+0.104592387 container create 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 (image=registry.fedoraproject.org/fedora:latest, name=strange_ptolemy, vendor=Fedora Project, version=37, name=fedora, license=MIT) time="2023-02-02T12:38:05Z" level=trace msg="{\"Id\":\"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\",\"Warnings\":[]}\n" API=response X-Reference-Id=0xc00056eb40 @ - - [02/Feb/2023:12:38:05 +0000] "POST /v4.3.1/libpod/containers/create HTTP/1.1" 201 88 "" "Go-http-client/1.1" time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:idle 2m+0h/4t connection(s)" X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:active 2m+0h/4t connection(s)" X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg="{\"httpproxy\":true,\"terminal\":true,\"stdin\":true,\"annotations\":{\"io.kubernetes.cri-o.TTY\":\"true\"},\"stop_timeout\":10,\"log_configuration\":{},\"raw_image_name\":\"fedora\",\"systemd\":\"true\",\"sdnotifyMode\":\"container\",\"pidns\":{},\"utsns\":{},\"remove\":true,\"containerCreateCommand\":[\"podman\",\"run\",\"--rm\",\"-ti\",\"fedora\"],\"init_container_type\":\"\",\"manage_password\":true,\"image\":\"fedora\",\"image_volume_mode\":\"anonymous\",\"ipcns\":{},\"volatile\":true,\"seccomp_policy\":\"default\",\"userns\":{},\"idmappings\":{\"HostUIDMapping\":true,\"HostGIDMapping\":true,\"UIDMap\":null,\"GIDMap\":null,\"AutoUserNs\":false,\"AutoUserNsOpts\":{\"Size\":0,\"InitialSize\":0,\"PasswdFile\":\"\",\"GroupFile\":\"\",\"AdditionalUIDMappings\":null,\"AdditionalGIDMappings\":null}},\"umask\":\"0022\",\"cgroupns\":{},\"cgroups_mode\":\"enabled\",\"netns\":{},\"Networks\":null,\"use_image_hosts\":false}" API=request X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=warning msg="Could not find mount at destination \"/etc/ssl/\" when parsing user volumes for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=warning msg="Could not find mount at destination \"/etc/pki/ca-trust/extracted/pem/\" when parsing user volumes for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=warning msg="Could not find mount at destination \"/etc/pki/tls/\" when parsing user volumes for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=debug msg="Looking up image \"19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\" in local containers storage" time="2023-02-02T12:38:05Z" level=debug msg="Trying \"19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\" ..." time="2023-02-02T12:38:05Z" level=debug msg="parsed reference into \"[overlay@/home/gitlab-runner/.local/share/containers/storage+/run/user/993/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\"" time="2023-02-02T12:38:05Z" level=debug msg="Found image \"19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\" as \"19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\" in local containers storage" time="2023-02-02T12:38:05Z" level=debug msg="Found image \"19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\" as \"19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\" in local containers storage ([overlay@/home/gitlab-runner/.local/share/containers/storage+/run/user/993/containers]@19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a)" time="2023-02-02T12:38:05Z" level=trace msg="{\"Id\":\"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\",\"Created\":\"2023-02-02T12:38:05.16826989Z\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"Path\":\"/bin/bash\",\"Args\":[\"/bin/bash\"],\"State\":{\"OciVersion\":\"1.0.2-dev\",\"Status\":\"created\",\"Running\":false,\"Paused\":false,\"Restarting\":false,\"OOMKilled\":false,\"Dead\":false,\"Pid\":0,\"ExitCode\":0,\"Error\":\"\",\"StartedAt\":\"0001-01-01T00:00:00Z\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"FinishedAt\":\"0001-01-01T00:00:00Z\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"Health\":{\"Status\":\"\",\"FailingStreak\":0,\"Log\":null},\"CheckpointedAt\":\"0001-01-01T00:00:00Z\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"RestoredAt\":\"0001-01-01T00:00:00Z\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg="},\"Image\":\"19c0ae4dd222b7c3b590fb11d7578105944c314ed90a6f125cf98d399fd14c4a\",\"ImageDigest\":\"sha256:ce08a91085403ecbc637eb2a96bd3554d75537871a12a14030b89243501050f2\",\"ImageName\":\"registry.fedoraproject.org/fedora:latest\",\"Rootfs\":\"\",\"Pod\":\"\",\"ResolvConfPath\":\"\",\"HostnamePath\":\"\",\"HostsPath\":\"\",\"StaticDir\":\"/home/gitlab-runner/.local/share/containers/storage/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata\",\"OCIRuntime\":\"crun\",\"ConmonPidFile\":\"/run/user/993/containers/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata/conmon.pid\",\"PidFile\":\"/run/user/993/containers/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata/pidfile\",\"Name\":\"strange_ptolemy\",\"RestartCount\":0,\"Driver\":\"overlay\",\"MountLabel\":\"system_u:object_r:container_file_t:s0:c106,c218\",\"ProcessLabel\":\"system_u:system_r:container_t:s0:c106,c218\",\"AppArmorProfile\":\"\",\"EffectiveCaps\":[\"CAP_CHOWN\",\"CAP_DAC_OVERRIDE\",\"CAP_FOWNER\",\"CAP_FSETID\",\"CAP_KILL\",\"CAP_NET_BIND_SERVICE\",\"CAP_SETFCAP\",\"CAP_SETGID\",\"CAP_SETPCAP\",\"CAP_SETUID\",\"CAP_SYS_CHROOT\"],\"BoundingCaps\":[\"CAP_CHOWN\",\"CAP_DAC_OVERRIDE\",\"CAP_FOWNER\",\"CAP_FSETID\",\"CAP_KILL\",\"CAP_NET_BIND_SERVICE\",\"CAP_SETFCAP\",\"CAP_SETGID\",\"CAP_SETPCAP\",\"CAP_SETUID\",\"CAP_SYS_CHROOT\"],\"ExecIDs\":[],\"GraphDriver\":{\"Name\":\"overlay\",\"Data\":{\"LowerDir\":\"/home/gitlab-runner/.local/share/containers/storage/overlay/ab03326cd6b0316148039cc3533a48126b41675046011565f840e042caab0cbf/diff\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"UpperDir\":\"/home/gitlab-runner/.local/share/containers/storage/overlay/5cd0b38ca5f16612df5565edf14a59beb16e5fde843933c0bd23954891aaeddf/diff\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"WorkDir\":\"/home/gitlab-runner/.local/share/containers/storage/overlay/5cd0b38ca5f16612df5565edf14a59beb16e5fde843933c0bd23954891aaeddf/work\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg="}},\"Mounts\":[],\"Dependencies\":[],\"NetworkSettings\":{\"EndpointID\":\"\",\"Gateway\":\"\",\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"MacAddress\":\"\",\"Bridge\":\"\",\"SandboxID\":\"\",\"HairpinMode\":false,\"LinkLocalIPv6Address\":\"\",\"LinkLocalIPv6PrefixLen\":0,\"Ports\":{},\"SandboxKey\":\"\"},\"Namespace\":\"\",\"IsInfra\":false,\"IsService\":false,\"Config\":{\"Hostname\":\"63d50e2e71e6\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":true,\"OpenStdin\":true,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"TERM=xterm\",\"container=oci\",\"DISTTAG=f37container\",\"FGC=f37\"],\"Cmd\":[\"/bin/bash\"],\"Image\":\"registry.fedoraproject.org/fedora:latest\",\"Volumes\":null,\"WorkingDir\":\"/\",\"Entrypoint\":\"\",\"OnBuild\":null,\"Labels\":{\"license\":\"MIT\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"name\":\"fedora\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"vendor\":\"Fedora Project\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"version\":\"37\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg="},\"Annotations\":{\"io.kubernetes.cri-o.TTY\":\"true\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"io.podman.annotations.autoremove\":\"TRUE\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"io.podman.annotations.init\":\"FALSE\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"io.podman.annotations.privileged\":\"FALSE\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg=",\"io.podman.annotations.publish-all\":\"FALSE\"" API=response X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=trace msg="},\"StopSignal\":15,\"HealthcheckOnFailureAction\":\"none\",\"CreateCommand\":[\"podman\",\"run\",\"--rm\",\"-ti\",\"fedora\"],\"Umask\":\"0022\",\"Timeout\":0,\"StopTimeout\":10,\"Passwd\":true,\"sdNotifyMode\":\"container\"},\"HostConfig\":{\"Binds\":[],\"CgroupManager\":\"systemd\",\"CgroupMode\":\"private\",\"ContainerIDFile\":\"\",\"LogConfig\":{\"Type\":\"journald\",\"Config\":null,\"Path\":\"\",\"Tag\":\"\",\"Size\":\"0B\"},\"NetworkMode\":\"slirp4netns\",\"PortBindings\":{},\"RestartPolicy\":{\"Name\":\"\",\"MaximumRetryCount\":0},\"AutoRemove\":true,\"VolumeDriver\":\"\",\"VolumesFrom\":null,\"CapAdd\":[],\"CapDrop\":[\"CAP_AUDIT_WRITE\",\"CAP_MKNOD\",\"CAP_NET_RAW\"],\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":[],\"GroupAdd\":[],\"IpcMode\":\"shareable\",\"Cgroup\":\"\",\"Cgroups\":\"default\",\"Links\":null,\"OomScoreAdj\":0,\"PidMode\":\"private\",\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyRootfs\":false,\"SecurityOpt\":[],\"Tmpfs\":{},\"UTSMode\":\"private\",\"UsernsMode\":\"\",\"ShmSize\":65536000,\"Runtime\":\"oci\",\"ConsoleSize\":[0,0],\"Isolation\":\"\",\"CpuShares\":0,\"Memory\":0,\"NanoCpus\":0,\"CgroupParent\":\"user.slice\",\"BlkioWeight\":0,\"BlkioWeightDevice\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteIOps\":null,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"Devices\":[],\"DiskQuota\":0,\"KernelMemory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":0,\"OomKillDisable\":false,\"PidsLimit\":2048,\"Ulimits\":[],\"CpuCount\":0,\"CpuPercent\":0,\"IOMaximumIOps\":0,\"IOMaximumBandwidth\":0,\"CgroupConf\":null}}\n" API=response X-Reference-Id=0xc00056eb40 @ - - [02/Feb/2023:12:38:05 +0000] "GET /v4.3.1/libpod/containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/json?size=false HTTP/1.1" 200 4951 "" "Go-http-client/1.1" time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:idle 2m+0h/4t connection(s)" X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:new 2m+0h/4t connection(s)" X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:active 2m+0h/5t connection(s)" X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=debug msg="[graphdriver] trying provided driver \"overlay\"" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that overlay is supported" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that overlay is supported" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that metacopy is not being used" time="2023-02-02T12:38:05Z" level=debug msg="backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that volatile is being used" time="2023-02-02T12:38:05Z" level=debug msg="overlay: mount_data=lowerdir=/home/gitlab-runner/.local/share/containers/storage/overlay/l/AJ2KTLAKESY3ROQNKYBLNSK7WB,upperdir=/home/gitlab-runner/.local/share/containers/storage/overlay/5cd0b38ca5f16612df5565edf14a59beb16e5fde843933c0bd23954891aaeddf/diff,workdir=/home/gitlab-runner/.local/share/containers/storage/overlay/5cd0b38ca5f16612df5565edf14a59beb16e5fde843933c0bd23954891aaeddf/work,,userxattr,volatile,context=\"system_u:object_r:container_file_t:s0:c106,c218\"" time="2023-02-02T12:38:05Z" level=debug msg="Made network namespace at /run/user/993/netns/netns-8c32bff7-6642-1a24-897b-96ecd8086ea7 for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=debug msg="Mounted container \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\" at \"/home/gitlab-runner/.local/share/containers/storage/overlay/5cd0b38ca5f16612df5565edf14a59beb16e5fde843933c0bd23954891aaeddf/merged\"" time="2023-02-02T12:38:05Z" level=debug msg="Created root filesystem for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 at /home/gitlab-runner/.local/share/containers/storage/overlay/5cd0b38ca5f16612df5565edf14a59beb16e5fde843933c0bd23954891aaeddf/merged" time="2023-02-02T12:38:05Z" level=debug msg="slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=48000 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /run/user/993/netns/netns-8c32bff7-6642-1a24-897b-96ecd8086ea7 tap0" time="2023-02-02T12:38:05Z" level=debug msg="found local resolver, using \"/run/systemd/resolve/resolv.conf\" to get the nameservers" time="2023-02-02T12:38:05Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]" time="2023-02-02T12:38:05Z" level=debug msg="/etc/system-fips does not exist on host, not mounting FIPS mode subscription" time="2023-02-02T12:38:05Z" level=debug msg="Setting Cgroups for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 to user.slice:libpod:63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=debug msg="reading hooks from /usr/share/containers/oci/hooks.d" 2023-02-02 12:38:05.149269632 +0000 UTC m=+0.065048469 image pull fedora time="2023-02-02T12:38:05Z" level=debug msg="Workdir \"/\" resolved to host path \"/home/gitlab-runner/.local/share/containers/storage/overlay/5cd0b38ca5f16612df5565edf14a59beb16e5fde843933c0bd23954891aaeddf/merged\"" time="2023-02-02T12:38:05Z" level=debug msg="Created OCI spec for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 at /home/gitlab-runner/.local/share/containers/storage/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata/config.json" time="2023-02-02T12:38:05Z" level=debug msg="running conmon: /usr/bin/conmon" args="[--api-version 1 -c 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 -u 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 -r /usr/bin/crun -b /home/gitlab-runner/.local/share/containers/storage/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata -p /run/user/993/containers/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata/pidfile -n strange_ptolemy --exit-dir /run/user/993/libpod/tmp/exits --full-attach -s -l journald --log-level trace --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/993/containers/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata/oci-log -t --conmon-pidfile /run/user/993/containers/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/gitlab-runner/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/993/containers --exit-command-arg --log-level --exit-command-arg trace --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/993/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/gitlab-runner/.local/share/containers/storage/volumes --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428]" time="2023-02-02T12:38:05Z" level=info msg="Running conmon under slice user.slice and unitName libpod-conmon-63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428.scope" time="2023-02-02T12:38:05Z" level=debug msg="Received: 122125" time="2023-02-02T12:38:05Z" level=info msg="Got Conmon PID as 122123" time="2023-02-02T12:38:05Z" level=debug msg="Created container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 in OCI runtime" 2023-02-02 12:38:05.289088502 +0000 UTC m=+0.204867328 container init 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 (image=registry.fedoraproject.org/fedora:latest, name=strange_ptolemy, license=MIT, name=fedora, vendor=Fedora Project, version=37) time="2023-02-02T12:38:05Z" level=info msg="Performing HTTP Hijack attach to container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" 2023-02-02 12:38:05.289534008 +0000 UTC m=+0.205312835 container attach 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 (image=registry.fedoraproject.org/fedora:latest, name=strange_ptolemy, license=MIT, name=fedora, vendor=Fedora Project, version=37) time="2023-02-02T12:38:05Z" level=debug msg="Successfully connected to container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 attach socket /home/gitlab-runner/.local/share/containers/storage/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata/attach" time="2023-02-02T12:38:05Z" level=debug msg="Going to hijack container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 attach connection" time="2023-02-02T12:38:05Z" level=info msg="Request Failed(Internal Server Error): unable to hijack connection" time="2023-02-02T12:38:05Z" level=trace msg="{\"cause\":\"unable to hijack connection\",\"message\":\"unable to hijack connection\",\"response\":500}\n" API=response X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=error msg="Error attaching to container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428: unable to hijack connection" time="2023-02-02T12:38:05Z" level=debug msg="Attach for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 completed successfully" @ - - [02/Feb/2023:12:38:05 +0000] "POST /v4.3.1/libpod/containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/attach?detachKeys=ctrl-p%2Cctrl-q&detachkeys=ctrl-p%2Cctrl-q&stderr=true&stdin=true&stdout=true&stream=true HTTP/1.1" 500 95 "" "Go-http-client/1.1" time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:idle 3m+0h/5t connection(s)" X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:active 3m+0h/5t connection(s)" X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=debug msg="Starting parallel job on container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=debug msg="Removing container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=debug msg="Cleaning up container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=debug msg="Tearing down network namespace at /run/user/993/netns/netns-8c32bff7-6642-1a24-897b-96ecd8086ea7 for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=debug msg="Successfully cleaned up container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=debug msg="Error unmounting /home/gitlab-runner/.local/share/containers/storage/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata/overlay/1096412638/merge with fusermount3 - exit status 1" time="2023-02-02T12:38:05Z" level=debug msg="Error unmounting /home/gitlab-runner/.local/share/containers/storage/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata/overlay/1371165858/merge with fusermount3 - exit status 1" time="2023-02-02T12:38:05Z" level=debug msg="Error unmounting /home/gitlab-runner/.local/share/containers/storage/overlay-containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428/userdata/overlay/1898062799/merge with fusermount3 - exit status 1" time="2023-02-02T12:38:05Z" level=debug msg="Unmounted container \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\"" time="2023-02-02T12:38:05Z" level=debug msg="Removing all exec sessions for container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428" time="2023-02-02T12:38:05Z" level=debug msg="Container 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 storage is already unmounted, skipping..." 2023-02-02 12:38:05.391671817 +0000 UTC m=+0.307450650 container remove 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428 (image=registry.fedoraproject.org/fedora:latest, name=strange_ptolemy, license=MIT, name=fedora, vendor=Fedora Project, version=37) time="2023-02-02T12:38:05Z" level=trace msg="[{\"Id\":\"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\",\"RawInput\":\"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\"}]\n" API=response X-Reference-Id=0xc00056e008 @ - - [02/Feb/2023:12:38:05 +0000] "DELETE /v4.3.1/libpod/containers/63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428?force=false&volumes=true HTTP/1.1" 200 154 "" "Go-http-client/1.1" time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:idle 3m+0h/5t connection(s)" X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:closed 3m+0h/5t connection(s)" X-Reference-Id=0xc00056e050 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:closed 2m+0h/5t connection(s)" X-Reference-Id=0xc00056eb40 time="2023-02-02T12:38:05Z" level=debug msg="IdleTracker:closed 1m+0h/5t connection(s)" X-Reference-Id=0xc00056e008 time="2023-02-02T12:38:05Z" level=debug msg="Called cleanup.PersistentPreRunE(/usr/bin/podman --root /home/gitlab-runner/.local/share/containers/storage --runroot /run/user/993/containers --log-level trace --cgroup-manager systemd --tmpdir /run/user/993/libpod/tmp --network-config-dir --network-backend netavark --volumepath /home/gitlab-runner/.local/share/containers/storage/volumes --runtime crun --storage-driver overlay --events-backend journald --syslog container cleanup --rm 63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428)" time="2023-02-02T12:38:05Z" level=trace msg="Reading configuration file \"/usr/share/containers/containers.conf\"" time="2023-02-02T12:38:05Z" level=debug msg="Merged system config \"/usr/share/containers/containers.conf\"" time="2023-02-02T12:38:05Z" level=trace msg="&{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.50.1 Annotations:[] BaseHostsFile: CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HostContainersInternalIP: HTTPProxy:true Init:false InitPath: IPCNS:shareable LogDriver:journald LogSizeMax:-1 LogTag: NetNS:private NoHosts:false PidsLimit:2048 PidNS:private PrepareVolumeOnCreate:false SeccompProfile: ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS: UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ConmonRsPath:[/usr/libexec/podman/conmonrs /usr/local/libexec/podman/conmonrs /usr/local/lib/podman/conmonrs /usr/bin/conmonrs /usr/sbin/conmonrs /usr/local/bin/conmonrs /usr/local/sbin/conmonrs /run/current-system/sw/bin/conmonrs] CompatAPIEnforceDockerHub:true DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath: EventsLogFileMaxSize:1000000 EventsLogger:journald graphRoot:/home/gitlab-runner/.local/share/containers/storage HelperBinariesDir:[/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: ImageVolumeMode:bind InfraCommand: InfraImage: InitPath:/usr/libexec/podman/catatonit LockType:shm MachineEnabled:false MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] krun:[/usr/bin/krun /usr/local/bin/krun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] runj:[/usr/local/bin/runj] runsc:[/usr/bin/runsc /usr/sbin/runsc /usr/local/bin/runsc /usr/local/sbin/runsc /bin/runsc /sbin/runsc /run/current-system/sw/bin/runsc]] PodExitPolicy:continue PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] SSHConfig:/home/gitlab-runner/.ssh/config RuntimePath:[] RuntimeSupportsJSON:[crun runc kata runsc krun] RuntimeSupportsNoCgroups:[crun krun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc krun] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 ServiceTimeout:5 StaticDir:/home/gitlab-runner/.local/share/containers/storage/libpod StopTimeout:10 ExitCommandDelay:300 ImageCopyTmpDir:/var/tmp TmpDir:/run/user/993/libpod/tmp VolumePath:/home/gitlab-runner/.local/share/containers/storage/volumes VolumePluginTimeout:5 VolumePlugins:map[] ChownCopiedFiles:true CompressionFormat:} Machine:{CPUs:1 DiskSize:100 Image:testing Memory:2048 User:core Volumes:[$HOME:$HOME]} Network:{NetworkBackend: CNIPluginDirs:[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] DefaultNetwork:podman DefaultSubnet:10.88.0.0/16 DefaultSubnetPools:[{Base:172.18.0.64/26 Size:29} {Base:172.18.0.128/25 Size:29} {Base:10.92.0.0/14 Size:24} {Base:10.96.0.0/11 Size:24} {Base:10.128.0.0/9 Size:24}] NetworkConfigDir: DNSBindPort:0} Secrets:{Driver:file Opts:map[]} ConfigMaps:{Driver: Opts:map[]}}" time="2023-02-02T12:38:05Z" level=trace msg="Reading configuration file \"/etc/containers/containers.conf\"" time="2023-02-02T12:38:05Z" level=debug msg="Failed to decode the keys [\"storage\" \"storage.driver\"] from \"/etc/containers/containers.conf\"." time="2023-02-02T12:38:05Z" level=debug msg="Merged system config \"/etc/containers/containers.conf\"" time="2023-02-02T12:38:05Z" level=trace msg="&{Containers:{Devices:[] Volumes:[/etc/ssl/:/etc/ssl/:O /etc/pki/ca-trust/extracted/pem/:/etc/pki/ca-trust/extracted/pem/:O /etc/pki/tls/:/etc/pki/tls/:O] ApparmorProfile:containers-default-0.50.1 Annotations:[] BaseHostsFile: CgroupNS:private Cgroups:enabled DefaultCapabilities:[CHOWN DAC_OVERRIDE FOWNER FSETID KILL NET_BIND_SERVICE SETFCAP SETGID SETPCAP SETUID SYS_CHROOT] DefaultSysctls:[net.ipv4.ping_group_range=0 0] DefaultUlimits:[] DefaultMountsFile: DNSServers:[10.0.2.3] DNSOptions:[edns0] DNSSearches:[] EnableKeyring:true EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm] EnvHost:false HostContainersInternalIP: HTTPProxy:true Init:false InitPath: IPCNS:shareable LogDriver:journald LogSizeMax:-1 LogTag: NetNS:private NoHosts:false PidsLimit:2048 PidNS:private PrepareVolumeOnCreate:false SeccompProfile: ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS: UserNSSize:65536} Engine:{CgroupCheck:false CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ConmonRsPath:[/usr/libexec/podman/conmonrs /usr/local/libexec/podman/conmonrs /usr/local/lib/podman/conmonrs /usr/bin/conmonrs /usr/sbin/conmonrs /usr/local/bin/conmonrs /usr/local/sbin/conmonrs /run/current-system/sw/bin/conmonrs] CompatAPIEnforceDockerHub:true DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath: EventsLogFileMaxSize:1000000 EventsLogger:journald graphRoot:/home/gitlab-runner/.local/share/containers/storage HelperBinariesDir:[/usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman] HooksDir:[/usr/share/containers/oci/hooks.d] ImageBuildFormat:oci ImageDefaultTransport:docker:// ImageParallelCopies:0 ImageDefaultFormat: ImageVolumeMode:bind InfraCommand: InfraImage: InitPath:/usr/libexec/podman/catatonit LockType:shm MachineEnabled:false MultiImageArchive:false Namespace: NetworkCmdPath: NetworkCmdOptions:[mtu=48000 enable_ipv6=false] NoPivotRoot:false NumLocks:2048 OCIRuntime:crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] krun:[/usr/bin/krun /usr/local/bin/krun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc] runj:[/usr/local/bin/runj] runsc:[/usr/bin/runsc /usr/sbin/runsc /usr/local/bin/runsc /usr/local/sbin/runsc /bin/runsc /sbin/runsc /run/current-system/sw/bin/runsc]] PodExitPolicy:continue PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] SSHConfig:/home/gitlab-runner/.ssh/config RuntimePath:[] RuntimeSupportsJSON:[crun runc kata runsc krun] RuntimeSupportsNoCgroups:[crun krun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc krun] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 ServiceTimeout:5 StaticDir:/home/gitlab-runner/.local/share/containers/storage/libpod StopTimeout:10 ExitCommandDelay:300 ImageCopyTmpDir:/var/tmp TmpDir:/run/user/993/libpod/tmp VolumePath:/home/gitlab-runner/.local/share/containers/storage/volumes VolumePluginTimeout:5 VolumePlugins:map[] ChownCopiedFiles:true CompressionFormat:} Machine:{CPUs:1 DiskSize:100 Image:testing Memory:2048 User:core Volumes:[$HOME:$HOME]} Network:{NetworkBackend: CNIPluginDirs:[/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] DefaultNetwork:podman DefaultSubnet:172.18.0.1/26 DefaultSubnetPools:[{Base:172.18.0.64/26 Size:29} {Base:172.18.0.128/25 Size:29}] NetworkConfigDir: DNSBindPort:0} Secrets:{Driver:file Opts:map[]} ConfigMaps:{Driver: Opts:map[]}}" time="2023-02-02T12:38:05Z" level=debug msg="Using conmon: \"/usr/bin/conmon\"" time="2023-02-02T12:38:05Z" level=debug msg="Initializing boltdb state at /home/gitlab-runner/.local/share/containers/storage/libpod/bolt_state.db" time="2023-02-02T12:38:05Z" level=debug msg="systemd-logind: Unknown object '/'." time="2023-02-02T12:38:05Z" level=debug msg="Using graph driver overlay" time="2023-02-02T12:38:05Z" level=debug msg="Using graph root /home/gitlab-runner/.local/share/containers/storage" time="2023-02-02T12:38:05Z" level=debug msg="Using run root /run/user/993/containers" time="2023-02-02T12:38:05Z" level=debug msg="Using static dir /home/gitlab-runner/.local/share/containers/storage/libpod" time="2023-02-02T12:38:05Z" level=debug msg="Using tmp dir /run/user/993/libpod/tmp" time="2023-02-02T12:38:05Z" level=debug msg="Using volume path /home/gitlab-runner/.local/share/containers/storage/volumes" time="2023-02-02T12:38:05Z" level=debug msg="Set libpod namespace to \"\"" time="2023-02-02T12:38:05Z" level=debug msg="[graphdriver] trying provided driver \"overlay\"" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that overlay is supported" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that overlay is supported" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that metacopy is not being used" time="2023-02-02T12:38:05Z" level=debug msg="Cached value indicated that native-diff is usable" time="2023-02-02T12:38:05Z" level=debug msg="backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false" time="2023-02-02T12:38:05Z" level=debug msg="Initializing event backend journald" time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument" time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument" time="2023-02-02T12:38:05Z" level=trace msg="found runtime \"/usr/bin/crun\"" time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument" time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument" time="2023-02-02T12:38:05Z" level=debug msg="Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument" time="2023-02-02T12:38:05Z" level=debug msg="Using OCI runtime \"/usr/bin/crun\"" time="2023-02-02T12:38:05Z" level=info msg="Setting parallel job count to 13" time="2023-02-02T12:38:05Z" level=debug msg="Error looking up container \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\": no container with name or ID \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\" found: no such container" time="2023-02-02T12:38:05Z" level=error msg="Running container cleanup: no container with name or ID \"63d50e2e71e6603d10084ac484fee631d68416fbd03b6b32b7e6db8021de0428\" found: no such container" ```

Describe the results you expected

Container starts using remote server like it starts when running directly.

podman info output

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpuset
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-1.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 98.62
    systemPercent: 0.4
    userPercent: 0.98
  cpus: 4
  distribution:
    distribution: fedora
    variant: cloud
    version: "37"
  eventLogger: journald
  hostname: <cut>
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 993
      size: 1
    - container_id: 1
      host_id: 200000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 993
      size: 1
    - container_id: 1
      host_id: 200000
      size: 65536
  kernel: 6.1.8-200.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 563015680
  memTotal: 8329515008
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.7.2-3.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.7.2
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/user/993/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/993/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8328572928
  swapTotal: 8328835072
  uptime: 22h 57m 38.00s (Approximately 0.92 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors: <cut>
    Prefix: docker.io
    PullFromMirror: ""
  <cut>
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/gitlab-runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/gitlab-runner/.local/share/containers/storage
  graphRootAllocated: 52527345664
  graphRootUsed: 4723773440
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 11
  runRoot: /run/user/993/containers
  volumePath: /home/gitlab-runner/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668178887
  BuiltTime: Fri Nov 11 15:01:27 2022
  GitCommit: ""
  GoVersion: go1.19.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Podman 4.3.1 rootless on Fedora 37 running in a VM. SELinux enabled. Server configured through user-level systemd service triggered by user-level systemd socket (default config available with Fedora with log level changed).

Changes in containers.conf ```toml [containers] volumes = [ "/etc/ssl/:/etc/ssl/:O", "/etc/pki/ca-trust/extracted/pem/:/etc/pki/ca-trust/extracted/pem/:O", "/etc/pki/tls/:/etc/pki/tls/:O" ] dns_servers = ["10.0.2.3"] dns_options = [ "edns0", ] [network] default_subnet="172.18.0.1/26" default_subnet_pools = [ {base = "172.18.0.64/26", size = 29}, {base = "172.18.0.128/25", size = 29}, ] [engine] network_cmd_options = [ "mtu=48000", "enable_ipv6=false", ] ```

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

Agalin commented 1 year ago

Update: it's no longer latest upstream release as 4.4 got released today.

Agalin commented 1 year ago

Tested with 4.4 and... it works! Will there be a backport to 4.3 or will 4.4 be released for Fedora 37?

vrothberg commented 1 year ago

Thanks for reaching out and testing, @Agalin.

Podman 4.4 will be released to Fedora 37 soon. Since it's already fixed, I am going to close the issue. Again, thanks for checking!

hsnoil commented 1 year ago

@vrothberg I ran into the same issue on 4.4.2

$ podman system service --time=0 --log-level=trace
$ CONTAINER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock podman run --rm -ti busybox
Error: unable to hijack connection

Here is my entire log

 ``` DEBU[0023] IdleTracker:new 0m+0h/0t connection(s) X-Reference-Id=0xc00059e028 DEBU[0023] IdleTracker:active 0m+0h/1t connection(s) X-Reference-Id=0xc00059e028 time="2023-03-12T03:32:11-04:00" level=trace msg=OK API=response X-Reference-Id=0xc00059e028 @ - - [12/Mar/2023:03:32:11 -0400] "GET /v4.4.2/libpod/_ping HTTP/1.1" 200 2 "" "Go-http-client/1.1" DEBU[0023] IdleTracker:idle 1m+0h/1t connection(s) X-Reference-Id=0xc00059e028 DEBU[0023] IdleTracker:closed 1m+0h/1t connection(s) X-Reference-Id=0xc00059e028 DEBU[0023] IdleTracker:new 0m+0h/1t connection(s) X-Reference-Id=0xc00059e048 DEBU[0023] IdleTracker:active 0m+0h/2t connection(s) X-Reference-Id=0xc00059e048 time="2023-03-12T03:32:11-04:00" level=trace msg=OK API=response X-Reference-Id=0xc00059e048 @ - - [12/Mar/2023:03:32:11 -0400] "GET /v4.4.2/libpod/_ping HTTP/1.1" 200 2 "" "Go-http-client/1.1" DEBU[0023] IdleTracker:idle 1m+0h/2t connection(s) X-Reference-Id=0xc00059e048 DEBU[0023] IdleTracker:closed 1m+0h/2t connection(s) X-Reference-Id=0xc00059e048 DEBU[0023] IdleTracker:new 0m+0h/2t connection(s) X-Reference-Id=0xc000144008 DEBU[0023] IdleTracker:active 0m+0h/3t connection(s) X-Reference-Id=0xc000144008 DEBU[0023] Successfully loaded network k3d-k3s-default: &{k3d-k3s-default cff136757ee9e10caec945f805e260bd5cd28202777024cca6236c9bdeef3751 bridge podman1 2023-03-11 02:42:35.11372516 -0500 EST [{{{10.89.0.0 ffffff00}} 10.89.0.1 }] false false true [] map[app:k3d] map[isolate:true] map[driver:host-local]} DEBU[0023] Successfully loaded 2 networks INFO[0023] Request Failed(Not Found): network not found time="2023-03-12T03:32:11-04:00" level=trace msg="{\"cause\":\"network not found\",\"message\":\"network not found\",\"response\":404}\n" API=response X-Reference-Id=0xc000144008 @ - - [12/Mar/2023:03:32:11 -0400] "GET /v4.4.2/libpod/networks/pasta/exists HTTP/1.1" 404 75 "" "Go-http-client/1.1" DEBU[0023] IdleTracker:idle 1m+0h/3t connection(s) X-Reference-Id=0xc000144008 DEBU[0023] IdleTracker:closed 1m+0h/3t connection(s) X-Reference-Id=0xc000144008 DEBU[0023] IdleTracker:new 0m+0h/3t connection(s) X-Reference-Id=0xc000144018 DEBU[0023] IdleTracker:active 0m+0h/4t connection(s) X-Reference-Id=0xc000144018 DEBU[0023] Pulling image busybox (policy: missing) DEBU[0023] Looking up image "busybox" in local containers storage DEBU[0023] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0023] Loading registries configuration "/etc/containers/registries.conf" DEBU[0023] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" DEBU[0023] Loading registries configuration "/etc/containers/registries.conf.d/myregistry.conf" DEBU[0023] Trying "docker.io/library/busybox:latest" ... DEBU[0023] parsed reference into "[overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] Found image "busybox" as "docker.io/library/busybox:latest" in local containers storage DEBU[0023] Found image "busybox" as "docker.io/library/busybox:latest" in local containers storage ([overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c) DEBU[0023] exporting opaque data as blob "sha256:bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] Looking up image "docker.io/library/busybox:latest" in local containers storage DEBU[0023] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0023] Trying "docker.io/library/busybox:latest" ... DEBU[0023] parsed reference into "[overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] Found image "docker.io/library/busybox:latest" as "docker.io/library/busybox:latest" in local containers storage DEBU[0023] Found image "docker.io/library/busybox:latest" as "docker.io/library/busybox:latest" in local containers storage ([overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c) DEBU[0023] exporting opaque data as blob "sha256:bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" time="2023-03-12T03:32:11-04:00" level=trace msg="{\"images\":[\"bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c\"],\"id\":\"bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c\"}\n" API=response X-Reference-Id=0xc000144018 @ - - [12/Mar/2023:03:32:11 -0400] "POST /v4.4.2/libpod/images/pull?alltags=false&arch=&authfile=&os=&password=&policy=missing&quiet=false&reference=busybox&username=&variant= HTTP/1.1" 200 152 "" "Go-http-client/1.1" DEBU[0023] IdleTracker:idle 1m+0h/4t connection(s) X-Reference-Id=0xc000144018 DEBU[0023] IdleTracker:new 1m+0h/4t connection(s) X-Reference-Id=0xc000144a08 DEBU[0023] IdleTracker:active 1m+0h/5t connection(s) X-Reference-Id=0xc000144a08 DEBU[0023] Looking up image "busybox" in local containers storage DEBU[0023] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0023] Trying "docker.io/library/busybox:latest" ... DEBU[0023] parsed reference into "[overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] Found image "busybox" as "docker.io/library/busybox:latest" in local containers storage DEBU[0023] Found image "busybox" as "docker.io/library/busybox:latest" in local containers storage ([overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c) DEBU[0023] exporting opaque data as blob "sha256:bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] Inspecting image bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c DEBU[0023] exporting opaque data as blob "sha256:bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] exporting opaque data as blob "sha256:bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] parsed reference into "[overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@ff9ced6cc0710b9560493044c17e3edd01c9039f29c46d547270cb942cc72bfd" DEBU[0023] exporting opaque data as blob "sha256:ff9ced6cc0710b9560493044c17e3edd01c9039f29c46d547270cb942cc72bfd" DEBU[0023] exporting opaque data as blob "sha256:bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" time="2023-03-12T03:32:11-04:00" level=trace msg="{\"Id\":\"bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c\",\"Digest\":\"sha256:c118f538365369207c12e5794c3cbfb7b042d950af590ae6c287ede74f29b7d4\",\"RepoTags\":[\"docker.io/library/busybox:latest\"],\"RepoDigests\":[\"docker.io/library/busybox@sha256:2c8ed5408179ff4f53242a4bdd2706110ce000be239fe37a61be9c52f704c437\",\"docker.io/library/busybox@sha256:c118f538365369207c12e5794c3cbfb7b042d950af590ae6c287ede74f29b7d4\"],\"Parent\":\"\",\"Comment\":\"\",\"Created\":\"2023-03-06T23:19:25.955224807Z\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg=",\"Config\":{\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"sh\"]},\"Version\":\"20.10.23\",\"Author\":\"\",\"Architecture\":\"amd64\",\"Os\":\"linux\",\"Size\":5097160,\"VirtualSize\":5097160,\"GraphDriver\":{\"Name\":\"overlay\",\"Data\":{\"UpperDir\":\"/home/test1/.local/share/containers/storage/overlay/427701cb9c9623cab51dcec9cfd66d7a88d189a14961120f4b2c62ce2e439b9d/diff\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg=",\"WorkDir\":\"/home/test1/.local/share/containers/storage/overlay/427701cb9c9623cab51dcec9cfd66d7a88d189a14961120f4b2c62ce2e439b9d/work\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg="}},\"RootFS\":{\"Type\":\"layers\",\"Layers\":[\"sha256:427701cb9c9623cab51dcec9cfd66d7a88d189a14961120f4b2c62ce2e439b9d\"]},\"Labels\":null,\"Annotations\":{},\"ManifestType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"User\":\"\",\"History\":[{\"created\":\"2023-03-06T23:19:25.827755207Z\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg=",\"created_by\":\"/bin/sh -c #(nop) ADD file:10c68f24723166c5dca276ce56a44db80a195a11377deeb04d816902b23608c2 in / \"},{\"created\":\"2023-03-06T23:19:25.955224807Z\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg=",\"created_by\":\"/bin/sh -c #(nop) CMD [\\\"sh\\\"]\",\"empty_layer\":true}],\"NamesHistory\":[\"docker.io/library/busybox:latest\"]}\n" API=response X-Reference-Id=0xc000144a08 @ - - [12/Mar/2023:03:32:11 -0400] "GET /v4.4.2/libpod/images/busybox/json?size=false HTTP/1.1" 200 1562 "" "Go-http-client/1.1" DEBU[0023] IdleTracker:idle 2m+0h/5t connection(s) X-Reference-Id=0xc000144a08 DEBU[0023] IdleTracker:active 2m+0h/5t connection(s) X-Reference-Id=0xc000144a08 DEBU[0023] Looking up image "busybox" in local containers storage DEBU[0023] Normalized platform linux/amd64 to {amd64 linux [] } DEBU[0023] Trying "docker.io/library/busybox:latest" ... DEBU[0023] parsed reference into "[overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] Found image "busybox" as "docker.io/library/busybox:latest" in local containers storage DEBU[0023] Found image "busybox" as "docker.io/library/busybox:latest" in local containers storage ([overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c) DEBU[0023] exporting opaque data as blob "sha256:bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] Inspecting image bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c DEBU[0023] exporting opaque data as blob "sha256:bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] exporting opaque data as blob "sha256:bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] Inspecting image bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c DEBU[0023] Inspecting image bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c DEBU[0023] Inspecting image bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c DEBU[0023] using systemd mode: false DEBU[0023] No hostname set; container's hostname will default to runtime default DEBU[0023] Loading seccomp profile from "/etc/containers/seccomp.json" DEBU[0023] Allocated lock 2 for container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 DEBU[0023] parsed reference into "[overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] exporting opaque data as blob "sha256:bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] Cached value indicated that idmapped mounts for overlay are not supported DEBU[0023] Check for idmapped mounts support DEBU[0023] Created container "e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6" DEBU[0023] Container "e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6" has work directory "/home/test1/.local/share/containers/storage/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata" DEBU[0023] Container "e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6" has run directory "/run/user/1000/containers/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata" time="2023-03-12T03:32:11-04:00" level=trace msg="{\"Id\":\"e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6\",\"Warnings\":[]}\n" API=response X-Reference-Id=0xc000144a08 @ - - [12/Mar/2023:03:32:11 -0400] "POST /v4.4.2/libpod/containers/create HTTP/1.1" 201 88 "" "Go-http-client/1.1" DEBU[0023] IdleTracker:idle 2m+0h/5t connection(s) X-Reference-Id=0xc000144a08 DEBU[0023] IdleTracker:active 2m+0h/5t connection(s) X-Reference-Id=0xc000144a08 DEBU[0023] Looking up image "bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" in local containers storage DEBU[0023] Trying "bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" ... DEBU[0023] parsed reference into "[overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" DEBU[0023] Found image "bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" as "bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" in local containers storage DEBU[0023] Found image "bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" as "bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c" in local containers storage ([overlay@/home/test1/.local/share/containers/storage+/run/user/1000/containers]@bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c) time="2023-03-12T03:32:11-04:00" level=trace msg="{\"Id\":\"e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6\",\"Created\":\"2023-03-12T03:32:11.241267612-04:00\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg=",\"Path\":\"sh\",\"Args\":[\"sh\"],\"State\":{\"OciVersion\":\"1.0.2-dev\",\"Status\":\"created\",\"Running\":false,\"Paused\":false,\"Restarting\":false,\"OOMKilled\":false,\"Dead\":false,\"Pid\":0,\"ExitCode\":0,\"Error\":\"\",\"StartedAt\":\"0001-01-01T00:00:00Z\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg=",\"FinishedAt\":\"0001-01-01T00:00:00Z\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg=",\"Health\":{\"Status\":\"\",\"FailingStreak\":0,\"Log\":null},\"CheckpointedAt\":\"0001-01-01T00:00:00Z\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg=",\"RestoredAt\":\"0001-01-01T00:00:00Z\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg="},\"Image\":\"bab98d58e29e4eb5744a69057a7b3d6ce99b19363a0e52c40301a5db43abf83c\",\"ImageDigest\":\"sha256:c118f538365369207c12e5794c3cbfb7b042d950af590ae6c287ede74f29b7d4\",\"ImageName\":\"docker.io/library/busybox:latest\",\"Rootfs\":\"\",\"Pod\":\"\",\"ResolvConfPath\":\"\",\"HostnamePath\":\"\",\"HostsPath\":\"\",\"StaticDir\":\"/home/test1/.local/share/containers/storage/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata\",\"OCIRuntime\":\"crun\",\"ConmonPidFile\":\"/run/user/1000/containers/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata/conmon.pid\",\"PidFile\":\"/run/user/1000/containers/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata/pidfile\",\"Name\":\"festive_shannon\",\"RestartCount\":0,\"Driver\":\"overlay\",\"MountLabel\":\"\",\"ProcessLabel\":\"\",\"AppArmorProfile\":\"\",\"EffectiveCaps\":[\"CAP_CHOWN\",\"CAP_DAC_OVERRIDE\",\"CAP_FOWNER\",\"CAP_FSETID\",\"CAP_KILL\",\"CAP_NET_BIND_SERVICE\",\"CAP_SETFCAP\",\"CAP_SETGID\",\"CAP_SETPCAP\",\"CAP_SETUID\",\"CAP_SYS_CHROOT\"],\"BoundingCaps\":[\"CAP_CHOWN\",\"CAP_DAC_OVERRIDE\",\"CAP_FOWNER\",\"CAP_FSETID\",\"CAP_KILL\",\"CAP_NET_BIND_SERVICE\",\"CAP_SETFCAP\",\"CAP_SETGID\",\"CAP_SETPCAP\",\"CAP_SETUID\",\"CAP_SYS_CHROOT\"],\"ExecIDs\":[],\"GraphDriver\":{\"Name\":\"overlay\",\"Data\":{\"LowerDir\":\"/home/test1/.local/share/containers/storage/overlay/427701cb9c9623cab51dcec9cfd66d7a88d189a14961120f4b2c62ce2e439b9d/diff\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg=",\"UpperDir\":\"/home/test1/.local/share/containers/storage/overlay/800778dfb21281bc290aa85af5c9e69b1015eea3584b6f8105c420ee0497d1cf/diff\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg=",\"WorkDir\":\"/home/test1/.local/share/containers/storage/overlay/800778dfb21281bc290aa85af5c9e69b1015eea3584b6f8105c420ee0497d1cf/work\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg="}},\"Mounts\":[],\"Dependencies\":[],\"NetworkSettings\":{\"EndpointID\":\"\",\"Gateway\":\"\",\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"MacAddress\":\"\",\"Bridge\":\"\",\"SandboxID\":\"\",\"HairpinMode\":false,\"LinkLocalIPv6Address\":\"\",\"LinkLocalIPv6PrefixLen\":0,\"Ports\":{},\"SandboxKey\":\"\"},\"Namespace\":\"\",\"IsInfra\":false,\"IsService\":false,\"Config\":{\"Hostname\":\"e42de8476903\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":true,\"OpenStdin\":true,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"TERM=xterm\",\"container=podman\"],\"Cmd\":[\"sh\"],\"Image\":\"docker.io/library/busybox:latest\",\"Volumes\":null,\"WorkingDir\":\"/\",\"Entrypoint\":\"\",\"OnBuild\":null,\"Labels\":null,\"Annotations\":{\"io.podman.annotations.autoremove\":\"TRUE\"" API=response X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:11-04:00" level=trace msg="},\"StopSignal\":15,\"HealthcheckOnFailureAction\":\"none\",\"CreateCommand\":[\"podman\",\"run\",\"--rm\",\"-ti\",\"busybox\"],\"Umask\":\"0022\",\"Timeout\":0,\"StopTimeout\":10,\"Passwd\":true,\"sdNotifyMode\":\"container\"},\"HostConfig\":{\"Binds\":[],\"CgroupManager\":\"systemd\",\"CgroupMode\":\"private\",\"ContainerIDFile\":\"\",\"LogConfig\":{\"Type\":\"journald\",\"Config\":null,\"Path\":\"\",\"Tag\":\"\",\"Size\":\"0B\"},\"NetworkMode\":\"slirp4netns\",\"PortBindings\":{},\"RestartPolicy\":{\"Name\":\"\",\"MaximumRetryCount\":0},\"AutoRemove\":true,\"VolumeDriver\":\"\",\"VolumesFrom\":null,\"CapAdd\":[],\"CapDrop\":[],\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":[],\"GroupAdd\":[],\"IpcMode\":\"shareable\",\"Cgroup\":\"\",\"Cgroups\":\"default\",\"Links\":null,\"OomScoreAdj\":0,\"PidMode\":\"private\",\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyRootfs\":false,\"SecurityOpt\":[],\"Tmpfs\":{},\"UTSMode\":\"private\",\"UsernsMode\":\"\",\"ShmSize\":65536000,\"Runtime\":\"oci\",\"ConsoleSize\":[0,0],\"Isolation\":\"\",\"CpuShares\":0,\"Memory\":0,\"NanoCpus\":0,\"CgroupParent\":\"user.slice\",\"BlkioWeight\":0,\"BlkioWeightDevice\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteIOps\":null,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"Devices\":[],\"DiskQuota\":0,\"KernelMemory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":0,\"OomKillDisable\":false,\"PidsLimit\":2048,\"Ulimits\":[],\"CpuCount\":0,\"CpuPercent\":0,\"IOMaximumIOps\":0,\"IOMaximumBandwidth\":0,\"CgroupConf\":null}}\n" API=response X-Reference-Id=0xc000144a08 @ - - [12/Mar/2023:03:32:11 -0400] "GET /v4.4.2/libpod/containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/json?size=false HTTP/1.1" 200 4502 "" "Go-http-client/1.1" DEBU[0023] IdleTracker:idle 2m+0h/5t connection(s) X-Reference-Id=0xc000144a08 DEBU[0023] IdleTracker:new 2m+0h/5t connection(s) X-Reference-Id=0xc0000113d0 DEBU[0023] IdleTracker:active 2m+0h/6t connection(s) X-Reference-Id=0xc0000113d0 DEBU[0023] Cached value indicated that volatile is being used DEBU[0023] overlay: mount_data=lowerdir=/home/test1/.local/share/containers/storage/overlay/l/ENXBDQN66S4FY64OOSJODFZ2ER,upperdir=/home/test1/.local/share/containers/storage/overlay/800778dfb21281bc290aa85af5c9e69b1015eea3584b6f8105c420ee0497d1cf/diff,workdir=/home/test1/.local/share/containers/storage/overlay/800778dfb21281bc290aa85af5c9e69b1015eea3584b6f8105c420ee0497d1cf/work,,userxattr,volatile DEBU[0023] Made network namespace at /run/user/1000/netns/netns-4cd3c76d-f917-34a1-18b1-35f69e91f7c1 for container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 DEBU[0023] Mounted container "e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6" at "/home/test1/.local/share/containers/storage/overlay/800778dfb21281bc290aa85af5c9e69b1015eea3584b6f8105c420ee0497d1cf/merged" DEBU[0023] Created root filesystem for container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 at /home/test1/.local/share/containers/storage/overlay/800778dfb21281bc290aa85af5c9e69b1015eea3584b6f8105c420ee0497d1cf/merged DEBU[0023] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/netns-4cd3c76d-f917-34a1-18b1-35f69e91f7c1 tap0 DEBU[0023] Skipping unrecognized mount in /etc/containers/mounts.conf: "# This configuration file specifies the default mounts for each container of the" DEBU[0023] Skipping unrecognized mount in /etc/containers/mounts.conf: "# tools adhering to this file (e.g., CRI-O, Podman, Buildah). The format of the" DEBU[0023] Skipping unrecognized mount in /etc/containers/mounts.conf: "# config is /SRC:/DST, one mount per line." DEBU[0023] /etc/system-fips does not exist on host, not mounting FIPS mode subscription DEBU[0023] Setting Cgroups for container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 to user.slice:libpod:e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 DEBU[0023] reading hooks from /usr/share/containers/oci/hooks.d DEBU[0023] Workdir "/" resolved to host path "/home/test1/.local/share/containers/storage/overlay/800778dfb21281bc290aa85af5c9e69b1015eea3584b6f8105c420ee0497d1cf/merged" DEBU[0023] Created OCI spec for container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 at /home/test1/.local/share/containers/storage/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata/config.json DEBU[0023] running conmon: /usr/bin/conmon args="[--api-version 1 -c e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 -u e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 -r /usr/bin/crun -b /home/test1/.local/share/containers/storage/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata -p /run/user/1000/containers/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata/pidfile -n festive_shannon --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l journald --log-level trace --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1000/containers/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata/oci-log -t --conmon-pidfile /run/user/1000/containers/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/test1/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg trace --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/test1/.local/share/containers/storage/volumes --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6]" INFO[0023] Running conmon under slice user.slice and unitName libpod-conmon-e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6.scope DEBU[0023] Received: 17572 INFO[0023] Got Conmon PID as 17566 DEBU[0023] Created container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 in OCI runtime INFO[0023] Performing HTTP Hijack attach to container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 DEBU[0023] Successfully connected to container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 attach socket /home/test1/.local/share/containers/storage/overlay-containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/userdata/attach DEBU[0023] Going to hijack container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 attach connection INFO[0023] Request Failed(Internal Server Error): unable to hijack connection time="2023-03-12T03:32:11-04:00" level=trace msg="{\"cause\":\"unable to hijack connection\",\"message\":\"unable to hijack connection\",\"response\":500}\n" API=response X-Reference-Id=0xc0000113d0 ERRO[0023] Error attaching to container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6: unable to hijack connection DEBU[0023] Attach for container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 completed successfully @ - - [12/Mar/2023:03:32:11 -0400] "POST /v4.4.2/libpod/containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6/attach?detachKeys=ctrl-p%2Cctrl-q&detachkeys=ctrl-p%2Cctrl-q&stderr=true&stdin=true&stdout=true&stream=true HTTP/1.1" 500 95 "" "Go-http-client/1.1" DEBU[0023] IdleTracker:idle 3m+0h/6t connection(s) X-Reference-Id=0xc0000113d0 DEBU[0023] IdleTracker:active 3m+0h/6t connection(s) X-Reference-Id=0xc0000113d0 DEBU[0023] Starting parallel job on container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 DEBU[0023] Removing container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 DEBU[0023] Cleaning up container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 DEBU[0023] Tearing down network namespace at /run/user/1000/netns/netns-4cd3c76d-f917-34a1-18b1-35f69e91f7c1 for container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 DEBU[0023] Successfully cleaned up container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 DEBU[0023] Unmounted container "e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6" DEBU[0023] Removing all exec sessions for container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 DEBU[0023] Container e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6 storage is already unmounted, skipping... time="2023-03-12T03:32:11-04:00" level=trace msg="[{\"Id\":\"e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6\"}]\n" API=response X-Reference-Id=0xc0000113d0 @ - - [12/Mar/2023:03:32:11 -0400] "DELETE /v4.4.2/libpod/containers/e42de8476903a2dd884d4a9c143c7b57080380e989e28173d52a900f13dfe4e6?force=false&volumes=true HTTP/1.1" 200 76 "" "Go-http-client/1.1" DEBU[0023] IdleTracker:idle 3m+0h/6t connection(s) X-Reference-Id=0xc0000113d0 DEBU[0023] IdleTracker:closed 3m+0h/6t connection(s) X-Reference-Id=0xc0000113d0 DEBU[0023] IdleTracker:closed 2m+0h/6t connection(s) X-Reference-Id=0xc000144018 DEBU[0023] IdleTracker:closed 1m+0h/6t connection(s) X-Reference-Id=0xc000144a08 time="2023-03-12T03:32:47-04:00" level=trace msg="{\"httpproxy\":true,\"terminal\":true,\"stdin\":true,\"stop_timeout\":10,\"log_configuration\":{},\"raw_image_name\":\"busybox\",\"systemd\":\"true\",\"sdnotifyMode\":\"container\",\"pidns\":{},\"utsns\":{},\"remove\":true,\"containerCreateCommand\":[\"podman\",\"run\",\"--rm\",\"-ti\",\"busybox\"],\"init_container_type\":\"\",\"manage_password\":true,\"image\":\"busybox\",\"image_volume_mode\":\"anonymous\",\"ipcns\":{},\"volatile\":true,\"seccomp_policy\":\"default\",\"userns\":{},\"idmappings\":{\"HostUIDMapping\":true,\"HostGIDMapping\":true,\"UIDMap\":null,\"GIDMap\":null,\"AutoUserNs\":false,\"AutoUserNsOpts\":{\"Size\":0,\"InitialSize\":0,\"PasswdFile\":\"\",\"GroupFile\":\"\",\"AdditionalUIDMappings\":null,\"AdditionalGIDMappings\":null}},\"umask\":\"0022\",\"cgroupns\":{},\"cgroups_mode\":\"enabled\",\"netns\":{},\"Networks\":null,\"use_image_hosts\":false}" API=request X-Reference-Id=0xc000144a08 ```

It works fine when I run it as:

 $ podman system service --time=0 --log-level=debug

Here is my podman info

 ``` host: arch: amd64 buildahVersion: 1.29.0 cgroupControllers: - memory - pids cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.1.5-2.1.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.5, commit: unknown' cpuUtilization: idlePercent: 94.16 systemPercent: 1.16 userPercent: 4.69 cpus: 8 distribution: distribution: '"opensuse-tumbleweed"' version: "20230308" eventLogger: journald hostname: localhost.localdomain idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 6.2.1-1-default linkmode: dynamic logDriver: journald memFree: 3546005504 memTotal: 12294696960 networkBackend: netavark ociRuntime: name: crun package: crun-1.8.1-1.1.x86_64 path: /usr/bin/crun version: |- crun version 1.8.1 commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30 rundir: /run/user/1000/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +YAJL os: linux remoteSocket: path: /run/user/1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /etc/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.0-1.1.x86_64 version: |- slirp4netns version 1.2.0 commit: unknown libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 5 libseccomp: 2.5.4 swapFree: 0 swapTotal: 0 uptime: 26h 38m 17.00s (Approximately 1.08 days) plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan volume: - local registries: localhost:44273: Blocked: false Insecure: true Location: localhost:44273 MirrorByDigestOnly: false Mirrors: null Prefix: localhost:44273 PullFromMirror: "" search: - registry.opensuse.org - registry.suse.com - docker.io store: configFile: /home/test1/.config/containers/storage.conf containerStore: number: 3 paused: 0 running: 3 stopped: 0 graphDriverName: overlay graphOptions: {} graphRoot: /home/test1/.local/share/containers/storage graphRootAllocated: 25732034560 graphRootUsed: 4838006784 graphStatus: Backing Filesystem: btrfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 79 runRoot: /run/user/1000/containers transientStore: false volumePath: /home/test1/.local/share/containers/storage/volumes version: APIVersion: 4.4.2 Built: 1677456000 BuiltTime: Sun Feb 26 19:00:00 2023 GitCommit: "" GoVersion: go1.18.10 Os: linux OsArch: linux/amd64 Version: 4.4.2 ```
vrothberg commented 1 year ago

@hsnoil, thanks for reaching out. Please open a new issue with the reproducer.