search

containers / prometheus-podman-exporter

Prometheus exporter for podman environments exposing containers, pods, images, volumes and networks information.
Apache License 2.0
138 stars 23 forks source link

SIGSEGV on startup until user runs "podman system reset" #230

Closed ruipin closed 2 months ago

ruipin commented 6 months ago

I have a few Proxmox machines running a multitude of VMs, each with multiple users running various podman containers.

The VMs themselves are running Debian bookworm.

Each user runs prometheus-podman-exporter under a systemd service:

[Unit]
Description=Prometheus Podman Exporter
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

StartLimitIntervalSec=300
StartLimitBurst=3

[Service]
Restart=on-failure
RestartSec=30

ExecStart=/bin/bash -c ' \
    exec /opt/prometheus/podman-exporter/prometheus-podman-exporter \
    --web.listen-address "127.0.0.1:$((40000 + %U))" \
    --collector.enable-all \
'

[Install]
WantedBy=default.target

This service is run by the various podman user accounts using systemctl --user. This works perfectly on many of my machines, but one of them segfaults whenever I start the service on any VM and any user:

ts=2024-04-25T18:46:14.302Z caller=exporter.go:68 level=info msg="Starting podman-prometheus-exporter" version="(version=1.11.0, branch=, revision=1)"
ts=2024-04-25T18:46:14.302Z caller=exporter.go:69 level=info msg=metrics enhanced=false
ts=2024-04-25T18:46:14.302Z caller=handler.go:94 level=info msg="enabled collectors"
ts=2024-04-25T18:46:14.302Z caller=handler.go:105 level=info collector=container
ts=2024-04-25T18:46:14.302Z caller=handler.go:105 level=info collector=image
ts=2024-04-25T18:46:14.302Z caller=handler.go:105 level=info collector=network
ts=2024-04-25T18:46:14.302Z caller=handler.go:105 level=info collector=pod
ts=2024-04-25T18:46:14.302Z caller=handler.go:105 level=info collector=system
ts=2024-04-25T18:46:14.302Z caller=handler.go:105 level=info collector=volume
ts=2024-04-25T18:46:14.306Z caller=events.go:17 level=debug msg="starting podman event streamer"
ts=2024-04-25T18:46:14.306Z caller=events.go:20 level=debug msg="update images"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xfaf217]

goroutine 1 [running]:
github.com/containers/common/libimage.(*Runtime).ListImages(0x0, {0x1c9a700, 0xc0003cbbf0}, {0x0?, 0xc00022d740?, 0x4176e5?}, 0x30?)
        /opt/prometheus/podman-exporter/v1.11.0/vendor/github.com/containers/common/libimage/runtime.go:587 +0x97
github.com/containers/podman/v5/pkg/domain/infra/abi.(*ImageEngine).List(0xc000178000, {0x1c9a700, 0xc0003cbbf0}, {0x0?, {0x0?, 0x763b80?, 0x27d8680?}})
        /opt/prometheus/podman-exporter/v1.11.0/vendor/github.com/containers/podman/v5/pkg/domain/infra/abi/images_list.go:25 +0x174
github.com/containers/prometheus-podman-exporter/pdcs.updateImages()
        /opt/prometheus/podman-exporter/v1.11.0/pdcs/image.go:48 +0x7c
github.com/containers/prometheus-podman-exporter/pdcs.StartEventStreamer({0x1c8b720?, 0xc00007f740}, 0x1)
        /opt/prometheus/podman-exporter/v1.11.0/pdcs/events.go:21 +0x1d3
github.com/containers/prometheus-podman-exporter/exporter.Start(0x0?, {0x0?, 0x0?, 0x0?})
        /opt/prometheus/podman-exporter/v1.11.0/exporter/exporter.go:93 +0x5fb
github.com/containers/prometheus-podman-exporter/cmd.run(0x27ebd40?, {0xc00007f480?, 0x4?, 0x19c8839?})
        /opt/prometheus/podman-exporter/v1.11.0/cmd/root.go:53 +0x1c
github.com/spf13/cobra.(*Command).execute(0x27ebd40, {0xc0000400b0, 0x4, 0x4})
        /opt/prometheus/podman-exporter/v1.11.0/vendor/github.com/spf13/cobra/command.go:987 +0xaa3
github.com/spf13/cobra.(*Command).ExecuteC(0x27ebd40)
        /opt/prometheus/podman-exporter/v1.11.0/vendor/github.com/spf13/cobra/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        /opt/prometheus/podman-exporter/v1.11.0/vendor/github.com/spf13/cobra/command.go:1039
github.com/containers/prometheus-podman-exporter/cmd.Execute()
        /opt/prometheus/podman-exporter/v1.11.0/cmd/root.go:61 +0x1e
main.main()
        /opt/prometheus/podman-exporter/v1.11.0/main.go:8 +0xf

The VMs are running podman 4.3.1.

I have tried both prometheus-podman-exporter v1.10.1 and v1.11.0. The packages were built using make clean && make binary using the debian golang package provided by bookworm-backports for go 1.21.8.

This happens no matter how many / which containers are running, and even if no containers are running. The segfault also happens independently of the user (even users who have not ever run a podman container), except for root - in which case everything works fine.

The proxmox machines, VMs and containers are for the most part provisioned through ansible, and as such the VMs should be virtually the same, just running different containers.

The issues go away once each user runs podman system reset once.

Apologies if this isn't enough information, but I have spent quite a bit of time trying to figure out what is wrong, and am wondering if you might have any ideas. I am happy to provide any extra information.

ruipin commented 6 months ago

Funnily enough, immediately after writing this I realised that running podman system prune on each individual VM/user affected seems to fix this issue. Rubber ducky debugging in action 😅

I suspect at some point something broke something internal to podman that this exporter relies on. Whatever this was affected all users and VMs provisioned on the same day, so presumably was related to my ansible script.

Leaving this issue open so that you can see it and maybe become aware that there might be some issue here, but feel free to close this issue if there's not enough information to reproduce.

ruipin commented 6 months ago

Just tried to provision a new user/service, and encountered this again.

It seems this is reproduceable for any newly created user on this specific machine, at least until I run podman system reset once.

navidys commented 6 months ago

Hi @ruipin

Have you tried to build from the master branch ? As far as I know podman v4 is not officially supported on debian.

navidys commented 6 months ago

Can u also attach podman system info output

ruipin commented 6 months ago

I'll give the master branch a try (v1.12.0-dev). Podman v4 is provided for Debian bookworm in the official debian repositories. I don't think it is unsupported? I've never had any issues with it, but maybe I'm missing something?

I've attached my podman system info output for one of the users: podman-system-info.txt

navidys commented 6 months ago

Hi @ruipin I've tried podman 4.3.1 under CentOS 8 stream but not facing your issue. Going to try on debian as well and will update you soon.

Can you run prometheus-podman-exporter binary directly intead of systemd service for new user and see if you are facing the issue ?

Regards

ruipin commented 6 months ago

Yes, I see the same crash even if I run directly, it does not matter if I run it through systemd, or the address/port I use, or whether I enable debug logging.

It is unfortunately not an easy one to reproduce, sorry.

It always happens when I create a new user until I do podman system reset, but I've only had it happen once on an existing user (that was previously working).

navidys commented 6 months ago

Hi @ruipin

I've installed Debian 12 (bookworm) on libvirt KVM with podman 4.3.1 and compiled the exporter (main branch) using golang 1.21.20 but still cannot simulate your issue even after creating a new user.

Are you using specific containers.conf for the new users or its a default one ?

ruipin commented 6 months ago

I am using a custom /etc/containers/containers.conf:

[containers]
# Use the k8s-file driver
# The default journald driver isn't a good fit for rootless containers and is discouraged by RedHat https://access.redhat.com/solutions/7009652
# With it, we can't use rootless 'podman logs' command
#log_driver = "k8s-file"

# Use the journald driver, so that we can see stdout/err when a container fails on start
# by running e.g. "sudo journalctl _SYSTEMD_USER_UNIT=<pod>:<container>.service"
log_driver = "journald"

# We can use the journald driver for events, as those are targeted at root
events_logger = "journald"

[networks]
# Change the default network range here
default_network = "podman"
default_subnet = "10.255.255.0/24"

I also use a custom /etc/containers/storage.conf:

[storage]
# Define the storage driver. We probably want 'overlay' or 'zfs'
driver = "overlay"

# Default paths, need to be explicitly set
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"
ruipin commented 6 months ago

I have tried creating a new user and doing the various steps of podman system reset manually until it works.

It seems stopping all containers, deleting /run/user/990/libpod, and then restarting the containers fixes the issue.

github-actions[bot] commented 5 months ago

A friendly reminder that this issue had no activity for 30 days.

K1kc4 commented 3 months ago

Having the same problem, podman system reset worked, and then after some time podman exporter crashed and i am again getting the same error. Installed podman via EPEL repo.

Exporter version: prometheus-podman-exporter (version=1.12.0, branch=main, revision=1.el9) Podman version: 4.9.4-rhel OS: RHEL 9.4

Using default containers.conf and storage.conf

navidys commented 3 months ago

Hi @K1kc4

Can u also attach the errors showing ?

Thanks

K1kc4 commented 3 months ago

Hello @navidys 

ts=2024-08-12T07:02:35.379Z caller=exporter.go:68 level=info msg="Starting podman-prometheus-exporter" version="(version=1.12.0, branch=main, revision=1.el9)"
ts=2024-08-12T07:02:35.379Z caller=exporter.go:69 level=info msg=metrics enhanced=false
ts=2024-08-12T07:02:35.379Z caller=handler.go:94 level=info msg="enabled collectors"
ts=2024-08-12T07:02:35.379Z caller=handler.go:105 level=info collector=container
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x90 pc=0x55c9106dd24f]

goroutine 1 [running]:
panic({0x55c910e51960?, 0x55c911b8fe50?})
        /usr/lib/golang/src/runtime/panic.go:1017 +0x3ac fp=0xc00052e508 sp=0xc00052e458 pc=0x55c90f78094c
runtime.panicmem(...)
        /usr/lib/golang/src/runtime/panic.go:261
runtime.sigpanic()
        /usr/lib/golang/src/runtime/signal_unix.go:861 +0x378 fp=0xc00052e568 sp=0xc00052e508 pc=0x55c90f797798
github.com/containers/podman/v5/libpod.(*Container).rwSize(0xc000428840?)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/libpod/container_internal.go:102 +0x2f fp=0xc00052e590 sp=0xc00052e568 pc=0x55c9106dd24f
github.com/containers/podman/v5/libpod.(*Container).RWSize(0x1000000000080?)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/libpod/container.go:1165 +0xc5 fp=0xc00052e638 sp=0xc00052e590 pc=0x55c9106bc545
github.com/containers/podman/v5/pkg/ps.ListContainerBatch.func1(0xc00002e280)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/pkg/ps/ps.go:226 +0xe97 fp=0xc00052e9a8 sp=0xc00052e638 pc=0x55c9107e5d57
github.com/containers/podman/v5/libpod.(*Container).Batch(0xc0002dc500, 0xc00052ec68)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/libpod/container_api.go:906 +0x136 fp=0xc00052ea08 sp=0xc00052e9a8 pc=0x55c9106c4976
github.com/containers/podman/v5/pkg/ps.ListContainerBatch(_, _, {0x1, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, ...})
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/pkg/ps/ps.go:153 +0x2f7 fp=0xc00052ef58 sp=0xc00052ea08 pc=0x55c9107e4417
github.com/containers/podman/v5/pkg/ps.GetContainerLists(0xc00034a340?, {0x1, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, 0x0, ...})
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/pkg/ps/ps.go:77 +0x5c8 fp=0xc00052f6e0 sp=0xc00052ef58 pc=0x55c9107e39e8
github.com/containers/podman/v5/pkg/domain/infra/abi.(*ContainerEngine).ContainerList(0x0?, {0x0?, 0x4?}, {0x1, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, ...})
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/pkg/domain/infra/abi/containers.go:1044 +0x58 fp=0xc00052f748 sp=0xc00052f6e0 pc=0x55c91080c218
github.com/containers/prometheus-podman-exporter/pdcs.updateContainerSize()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/pdcs/container.go:190 +0xf0 fp=0xc00052fa00 sp=0xc00052f748 pc=0x55c91084a4d0
github.com/containers/prometheus-podman-exporter/pdcs.StartCacheSizeTicker({0x55c911083268?, 0xc0004bb640}, 0xe10)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/pdcs/container.go:218 +0x1f8 fp=0xc00052fa98 sp=0xc00052fa00 pc=0x55c91084a898
github.com/containers/prometheus-podman-exporter/exporter.Start(0x0?, {0x0?, 0x0?, 0x0?})
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/exporter/exporter.go:94 +0x61c fp=0xc00052fbf8 sp=0xc00052fa98 pc=0x55c9108937fc
github.com/containers/prometheus-podman-exporter/cmd.run(0x55c911baea80?, {0x55c911c9b340?, 0x4?, 0x55c910987c3f?})
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/cmd/root.go:53 +0x1c fp=0xc00052fc58 sp=0xc00052fbf8 pc=0x55c91089607c
github.com/spf13/cobra.(*Command).execute(0x55c911baea80, {0xc000126050, 0x0, 0x0})
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/spf13/cobra/command.go:987 +0xaa3 fp=0xc00052fdf8 sp=0xc00052fc58 pc=0x55c9103fb863
github.com/spf13/cobra.(*Command).ExecuteC(0x55c911baea80)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/spf13/cobra/command.go:1115 +0x3ff fp=0xc00052fed0 sp=0xc00052fdf8 pc=0x55c9103fc13f
github.com/spf13/cobra.(*Command).Execute(...)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/spf13/cobra/command.go:1039
github.com/containers/prometheus-podman-exporter/cmd.Execute()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/cmd/root.go:61 +0x1e fp=0xc00052ff30 sp=0xc00052fed0 pc=0x55c9108961be
main.main()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/main.go:8 +0xf fp=0xc00052ff40 sp=0xc00052ff30 pc=0x55c91089682f
runtime.main()
        /usr/lib/golang/src/runtime/proc.go:267 +0x2d2 fp=0xc00052ffe0 sp=0xc00052ff40 pc=0x55c90f7838b2
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00052ffe8 sp=0xc00052ffe0 pc=0x55c90f7b4741

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000064fa8 sp=0xc000064f88 pc=0x55c90f783d0e
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:404
runtime.forcegchelper()
        /usr/lib/golang/src/runtime/proc.go:322 +0xb8 fp=0xc000064fe0 sp=0xc000064fa8 pc=0x55c90f783b98
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000064fe8 sp=0xc000064fe0 pc=0x55c90f7b4741
created by runtime.init.7 in goroutine 1
        /usr/lib/golang/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000065778 sp=0xc000065758 pc=0x55c90f783d0e
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
        /usr/lib/golang/src/runtime/mgcsweep.go:321 +0xdf fp=0xc0000657c8 sp=0xc000065778 pc=0x55c90f76ebff
runtime.gcenable.func1()
        /usr/lib/golang/src/runtime/mgc.go:200 +0x25 fp=0xc0000657e0 sp=0xc0000657c8 pc=0x55c90f763d25
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000657e8 sp=0xc0000657e0 pc=0x55c90f7b4741
created by runtime.gcenable in goroutine 1
        /usr/lib/golang/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00004c070?, 0x55c910c48790?, 0x0?, 0x0?, 0x0?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000065f70 sp=0xc000065f50 pc=0x55c90f783d0e
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x55c911c69200)
        /usr/lib/golang/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000065fa0 sp=0xc000065f70 pc=0x55c90f76c489
runtime.bgscavenge(0x0?)
        /usr/lib/golang/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000065fc8 sp=0xc000065fa0 pc=0x55c90f76ca39
runtime.gcenable.func2()
        /usr/lib/golang/src/runtime/mgc.go:201 +0x25 fp=0xc000065fe0 sp=0xc000065fc8 pc=0x55c90f763cc5
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x55c90f7b4741
created by runtime.gcenable in goroutine 1
        /usr/lib/golang/src/runtime/mgc.go:201 +0xa5

goroutine 18 [finalizer wait]:
runtime.gopark(0x0?, 0x55c91107bfe8?, 0xa0?, 0x65?, 0x2000000020?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000064620 sp=0xc000064600 pc=0x55c90f783d0e
runtime.runfinq()
        /usr/lib/golang/src/runtime/mfinal.go:193 +0x107 fp=0xc0000647e0 sp=0xc000064620 pc=0x55c90f762cc7
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000647e8 sp=0xc0000647e0 pc=0x55c90f7b4741
created by runtime.createfing in goroutine 1
        /usr/lib/golang/src/runtime/mfinal.go:163 +0x3d

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x94a7e59028881?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000060750 sp=0xc000060730 pc=0x55c90f783d0e
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000607e0 sp=0xc000060750 pc=0x55c90f7658e5
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x55c90f7b4741
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/lib/golang/src/runtime/mgc.go:1219 +0x1c

goroutine 5 [GC worker (idle)]:
runtime.gopark(0x94a7e59028a7f?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000066750 sp=0xc000066730 pc=0x55c90f783d0e
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1295 +0xe5 fp=0xc0000667e0 sp=0xc000066750 pc=0x55c90f7658e5
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000667e8 sp=0xc0000667e0 pc=0x55c90f7b4741
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/lib/golang/src/runtime/mgc.go:1219 +0x1c

goroutine 35 [select, locked to thread]:
runtime.gopark(0xc000061fa8?, 0x2?, 0x15?, 0xee?, 0xc000061fa4?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000061e38 sp=0xc000061e18 pc=0x55c90f783d0e
runtime.selectgo(0xc000061fa8, 0xc000061fa0, 0x0?, 0x0, 0xc000061f01?, 0x1)
        /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc000061f58 sp=0xc000061e38 pc=0x55c90f793ac5
runtime.ensureSigM.func1()
        /usr/lib/golang/src/runtime/signal_unix.go:1014 +0x1a5 fp=0xc000061fe0 sp=0xc000061f58 pc=0x55c90f7aafe5
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x55c90f7b4741
created by runtime.ensureSigM in goroutine 1
        /usr/lib/golang/src/runtime/signal_unix.go:997 +0xc8

goroutine 24 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
        /usr/lib/golang/src/runtime/lock_futex.go:236 +0x29 fp=0xc0004e97a0 sp=0xc0004e9768 pc=0x55c90f754dc9
os/signal.signal_recv()
        /usr/lib/golang/src/runtime/sigqueue.go:152 +0x29 fp=0xc0004e97c0 sp=0xc0004e97a0 pc=0x55c90f7b0fe9
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0004e97e0 sp=0xc0004e97c0 pc=0x55c90fbd4653
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004e97e8 sp=0xc0004e97e0 pc=0x55c90f7b4741
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/lib/golang/src/os/signal/signal.go:151 +0x1f

goroutine 36 [select]:
runtime.gopark(0xc0000627b0?, 0x2?, 0x0?, 0x0?, 0xc0000626ac?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000062538 sp=0xc000062518 pc=0x55c90f783d0e
runtime.selectgo(0xc0000627b0, 0xc0000626a8, 0x0?, 0x0, 0x0?, 0x1)
        /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc000062658 sp=0xc000062538 pc=0x55c90f793ac5
github.com/containers/podman/v5/libpod/shutdown.Start.func1()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/libpod/shutdown/handler.go:48 +0x87 fp=0xc0000627e0 sp=0xc000062658 pc=0x55c91059d0c7
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000627e8 sp=0xc0000627e0 pc=0x55c90f7b4741
created by github.com/containers/podman/v5/libpod/shutdown.Start in goroutine 1
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/libpod/shutdown/handler.go:47 +0xf1

goroutine 37 [select]:
runtime.gopark(0xc000062f88?, 0x2?, 0x0?, 0x0?, 0xc000062f84?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000062e30 sp=0xc000062e10 pc=0x55c90f783d0e
runtime.selectgo(0xc000062f88, 0xc000062f80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc000062f50 sp=0xc000062e30 pc=0x55c90f793ac5
database/sql.(*DB).connectionOpener(0xc0002cd1e0, {0x55c911092438, 0xc00048ed20})
        /usr/lib/golang/src/database/sql/sql.go:1218 +0x87 fp=0xc000062fb8 sp=0xc000062f50 pc=0x55c90ff45647
database/sql.OpenDB.func1()
        /usr/lib/golang/src/database/sql/sql.go:791 +0x28 fp=0xc000062fe0 sp=0xc000062fb8 pc=0x55c90ff43a68
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000062fe8 sp=0xc000062fe0 pc=0x55c90f7b4741
created by database/sql.OpenDB in goroutine 1
        /usr/lib/golang/src/database/sql/sql.go:791 +0x165

goroutine 46 [select]:
runtime.gopark(0xc000077d70?, 0x2?, 0x0?, 0x0?, 0xc000077cdc?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000077b38 sp=0xc000077b18 pc=0x55c90f783d0e
runtime.selectgo(0xc000077d70, 0xc000077cd8, 0xc0004b9a10?, 0x0, 0xf4240?, 0x1)
        /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc000077c58 sp=0xc000077b38 pc=0x55c90f793ac5
github.com/containers/podman/v5/libpod/events.EventLogFile.Read({{{0x55c910987c6b, 0x4}, {0xc0004b9a10, 0x2b}, 0xf4240}}, {0x55c9110922b0, 0x55c911c9b340}, {0xc0002c9c80, {0x55c911c9b340, 0x0, ...}, ...})
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/libpod/events/logfile.go:155 +0x485 fp=0xc000077dc8 sp=0xc000077c58 pc=0x55c9103d6e25
github.com/containers/podman/v5/libpod/events.(*EventLogFile).Read(0x0?, {0x55c9110922b0?, 0x55c911c9b340?}, {0xc0002c9c80, {0x55c911c9b340, 0x0, 0x0}, 0x0, {0x0, 0x0}, ...})
        <autogenerated>:1 +0xca fp=0xc000077e88 sp=0xc000077dc8 pc=0x55c9103d91ea
github.com/containers/podman/v5/libpod.(*Runtime).Events(...)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/libpod/events.go:174
github.com/containers/podman/v5/pkg/domain/infra/abi.(*ContainerEngine).Events(0x55c90f78ae3c?, {0x55c9110922b0?, 0x55c911c9b340?}, {0x0, 0xc0002c9c80, {0x55c911c9b340, 0x0, 0x0}, 0x1, {0x0, ...}, ...})
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/pkg/domain/infra/abi/events.go:12 +0x11a fp=0xc000077f50 sp=0xc000077e88 pc=0x55c91081397a
github.com/containers/prometheus-podman-exporter/pdcs.StartEventStreamer.func1()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/pdcs/events.go:31 +0x64 fp=0xc000077fe0 sp=0xc000077f50 pc=0x55c91084b1a4
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x55c90f7b4741
created by github.com/containers/prometheus-podman-exporter/pdcs.StartEventStreamer in goroutine 1
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/pdcs/events.go:30 +0x2c5

goroutine 45 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x1?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000063f08 sp=0xc000063ee8 pc=0x55c90f783d0e
runtime.chanrecv(0xc0002c9c20, 0xc000063fc8, 0x1)
        /usr/lib/golang/src/runtime/chan.go:583 +0x3cd fp=0xc000063f80 sp=0xc000063f08 pc=0x55c90f74f82d
runtime.chanrecv2(0x55c911c9b340?, 0x55c911092438?)
        /usr/lib/golang/src/runtime/chan.go:447 +0x12 fp=0xc000063fa8 sp=0xc000063f80 pc=0x55c90f74f452
github.com/containers/podman/v5/libpod.(*Runtime).startWorker.func1()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/libpod/runtime_worker.go:8 +0x6c fp=0xc000063fe0 sp=0xc000063fa8 pc=0x55c91077d5cc
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x55c90f7b4741
created by github.com/containers/podman/v5/libpod.(*Runtime).startWorker in goroutine 1
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/containers/podman/v5/libpod/runtime_worker.go:7 +0x8e

goroutine 29 [select]:
runtime.gopark(0xc000079db0?, 0x4?, 0x88?, 0x9c?, 0xc000079d68?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000079c10 sp=0xc000079bf0 pc=0x55c90f783d0e
runtime.selectgo(0xc000079db0, 0xc000079d60, 0x8dc08?, 0x0, 0x0?, 0x1)
        /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc000079d30 sp=0xc000079c10 pc=0x55c90f793ac5
github.com/nxadm/tail.(*Tail).waitForChanges(0xc0004da180)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/nxadm/tail/tail.go:386 +0x168 fp=0xc000079e00 sp=0xc000079d30 pc=0x55c9103cfd88
github.com/nxadm/tail.(*Tail).tailFileSync(0xc0004da180)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/nxadm/tail/tail.go:347 +0x4f6 fp=0xc000079fc8 sp=0xc000079e00 pc=0x55c9103cf936
github.com/nxadm/tail.TailFile.func1()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/nxadm/tail/tail.go:157 +0x25 fp=0xc000079fe0 sp=0xc000079fc8 pc=0x55c9103ceaa5
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x55c90f7b4741
created by github.com/nxadm/tail.TailFile in goroutine 46
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/nxadm/tail/tail.go:157 +0x2fa

goroutine 41 [chan receive]:
runtime.gopark(0x55c90f84d485?, 0x0?, 0xa8?, 0x77?, 0x55c90f84d3f9?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000067718 sp=0xc0000676f8 pc=0x55c90f783d0e
runtime.chanrecv(0xc000102660, 0x0, 0x1)
        /usr/lib/golang/src/runtime/chan.go:583 +0x3cd fp=0xc000067790 sp=0xc000067718 pc=0x55c90f74f82d
runtime.chanrecv1(0xc00048ee60?, 0x0?)
        /usr/lib/golang/src/runtime/chan.go:442 +0x12 fp=0xc0000677b8 sp=0xc000067790 pc=0x55c90f74f432
github.com/godbus/dbus/v5.newConn.func1()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/godbus/dbus/v5/conn.go:302 +0x2c fp=0xc0000677e0 sp=0xc0000677b8 pc=0x55c90fc35b2c
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000677e8 sp=0xc0000677e0 pc=0x55c90f7b4741
created by github.com/godbus/dbus/v5.newConn in goroutine 1
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/godbus/dbus/v5/conn.go:301 +0x4f6

goroutine 25 [IO wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc0004fba48 sp=0xc0004fba28 pc=0x55c90f783d0e
runtime.netpollblock(0x0?, 0xf74c966?, 0xc9?)
        /usr/lib/golang/src/runtime/netpoll.go:564 +0xf7 fp=0xc0004fba80 sp=0xc0004fba48 pc=0x55c90f77c617
internal/poll.runtime_pollWait(0x7ff054c52e28, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:343 +0x85 fp=0xc0004fbaa0 sp=0xc0004fba80 pc=0x55c90f7aea25
internal/poll.(*pollDesc).wait(0xc000195f80?, 0xc000014710?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0004fbac8 sp=0xc0004fbaa0 pc=0x55c90f8247a7
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).ReadMsg(0xc000195f80, {0xc000014710, 0x10, 0x10}, {0xc0004ec020, 0x1000, 0x1000}, 0x7ff054c0f148?)
        /usr/lib/golang/src/internal/poll/fd_unix.go:301 +0x38a fp=0xc0004fbbb8 sp=0xc0004fbac8 pc=0x55c90f826caa
net.(*netFD).readMsg(0xc000195f80, {0xc000014710?, 0x7ff054cca108?, 0x10?}, {0xc0004ec020?, 0xc0001fe5f0?, 0x0?}, 0xc0004fbce8?)
        /usr/lib/golang/src/net/fd_posix.go:78 +0x31 fp=0xc0004fbc40 sp=0xc0004fbbb8 pc=0x55c90f943d51
net.(*UnixConn).readMsg(0xc000128e38, {0xc000014710?, 0xc0004fbd88?, 0xc0004fbdc8?}, {0xc0004ec020?, 0x55c911c9b340?, 0xc0004f410d?})
        /usr/lib/golang/src/net/unixsock_posix.go:115 +0x45 fp=0xc0004fbcd0 sp=0xc0004fbc40 pc=0x55c90f965d65
net.(*UnixConn).ReadMsgUnix(0xc000128e38, {0xc000014710?, 0x55c9110922b0?, 0x55c911c9b340?}, {0xc0004ec020?, 0xc0001069c0?, 0x3?})
        /usr/lib/golang/src/net/unixsock.go:143 +0x36 fp=0xc0004fbd48 sp=0xc0004fbcd0 pc=0x55c90f963ab6
github.com/godbus/dbus/v5.(*oobReader).Read(0xc0004ec000, {0xc000014710?, 0x55c911b90a10?, 0x55c910db8260?})
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/godbus/dbus/v5/transport_unix.go:41 +0x3c fp=0xc0004fbdc0 sp=0xc0004fbd48 pc=0x55c90fc4d45c
io.ReadAtLeast({0x55c9110859e0, 0xc0004ec000}, {0xc000014710, 0x10, 0x10}, 0x10)
        /usr/lib/golang/src/io/io.go:335 +0x90 fp=0xc0004fbe08 sp=0xc0004fbdc0 pc=0x55c90f7fb470
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:354
github.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc0002ca720)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/godbus/dbus/v5/transport_unix.go:116 +0x22a fp=0xc0004fbf58 sp=0xc0004fbe08 pc=0x55c90fc4db4a
github.com/godbus/dbus/v5.(*Conn).inWorker(0xc0002de000)
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/godbus/dbus/v5/conn.go:389 +0x43 fp=0xc0004fbfc8 sp=0xc0004fbf58 pc=0x55c90fc35fe3
github.com/godbus/dbus/v5.(*Conn).Auth.func1()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/godbus/dbus/v5/auth.go:118 +0x25 fp=0xc0004fbfe0 sp=0xc0004fbfc8 pc=0x55c90fc335a5
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004fbfe8 sp=0xc0004fbfe0 pc=0x55c90f7b4741
created by github.com/godbus/dbus/v5.(*Conn).Auth in goroutine 1
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/godbus/dbus/v5/auth.go:118 +0x945

goroutine 30 [sleep]:
runtime.gopark(0x94a7e58dbe66f?, 0xc0004ea718?, 0xbe?, 0xfd?, 0x55c90f74f56b?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc0004ea6e8 sp=0xc0004ea6c8 pc=0x55c90f783d0e
time.Sleep(0xee6b280)
        /usr/lib/golang/src/runtime/time.go:195 +0x125 fp=0xc0004ea728 sp=0xc0004ea6e8 pc=0x55c90f7b16c5
github.com/nxadm/tail/watch.(*PollingFileWatcher).ChangeEvents.func1()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/nxadm/tail/watch/polling.go:69 +0x94 fp=0xc0004ea7e0 sp=0xc0004ea728 pc=0x55c9103ce294
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0004ea7e8 sp=0xc0004ea7e0 pc=0x55c90f7b4741
created by github.com/nxadm/tail/watch.(*PollingFileWatcher).ChangeEvents in goroutine 29
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/vendor/github.com/nxadm/tail/watch/polling.go:60 +0x19a

goroutine 47 [select]:
runtime.gopark(0xc000067f90?, 0x2?, 0x0?, 0x0?, 0xc000067f3c?)
        /usr/lib/golang/src/runtime/proc.go:398 +0xce fp=0xc000067dc8 sp=0xc000067da8 pc=0x55c90f783d0e
runtime.selectgo(0xc000067f90, 0xc000067f38, 0x0?, 0x0, 0x0?, 0x1)
        /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc000067ee8 sp=0xc000067dc8 pc=0x55c90f793ac5
github.com/containers/prometheus-podman-exporter/pdcs.StartEventStreamer.func2()
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/pdcs/events.go:39 +0xca fp=0xc000067fe0 sp=0xc000067ee8 pc=0x55c91084aeea
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000067fe8 sp=0xc000067fe0 pc=0x55c90f7b4741
created by github.com/containers/prometheus-podman-exporter/pdcs.StartEventStreamer in goroutine 1
        /builddir/build/BUILD/prometheus-podman-exporter-1.12.0/pdcs/events.go:37 +0x35f
Aborted (core dumped)
navidys commented 2 months ago

Hi @K1kc4

Can you let me know what version of podman you are using ?

I've tried with podman v5 and cannot simulate your issue.

can you also try latest release v1.13.0 of the exporter to see if you facing similar issue.

Regards

navidys commented 2 months ago

Hi

Tried podman v5.2.1 on Centos9 stream + prometheus-podman-exporter v1.13.0 and the issue is not happening.

Closing ticket.

Regards

Spectrik commented 2 weeks ago

We're having the same/similar problem. Detaiils:

Podman version: podman-4.9.4-13.0.1.module+el8.10.0+90412+9b361f34.x86_64

Exporter version: 1.10.1

System info:

Red Hat Enterprise Linux release 8.8 (Ootpa) Linux 5.15.0-300.163.18.1.1.el8uek.x86_64 #2 SMP Fri Oct 11 03:19:46 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Error:

ts=2024-10-23T13:35:15.005Z caller=exporter.go:68 level=info msg="Starting podman-prometheus-exporter" version="(version=1.10.1, branch=, revision=unknown)" 2024-10-23T06:35:15.005794-07:00 my-very-cool-host prometheus-podman-exporter[76891]: ts=2024-10-23T13:35:15.005Z caller=exporter.go:69 level=info msg=metrics enhanced=false 2024-10-23T06:35:15.005851-07:00 my-very-cool-host prometheus-podman-exporter[76891]: ts=2024-10-23T13:35:15.005Z caller=handler.go:94 level=info msg="enabled collectors" 2024-10-23T06:35:15.005899-07:00 my-very-cool-host prometheus-podman-exporter[76891]: ts=2024-10-23T13:35:15.005Z caller=handler.go:105 level=info collector=container 2024-10-23T06:35:15.005947-07:00 my-very-cool-host prometheus-podman-exporter[76891]: ts=2024-10-23T13:35:15.005Z caller=handler.go:105 level=info collector=image 2024-10-23T06:35:15.005995-07:00 my-very-cool-host prometheus-podman-exporter[76891]: ts=2024-10-23T13:35:15.005Z caller=handler.go:105 level=info collector=network 2024-10-23T06:35:15.006055-07:00 my-very-cool-host prometheus-podman-exporter[76891]: ts=2024-10-23T13:35:15.005Z caller=handler.go:105 level=info collector=pod 2024-10-23T06:35:15.006103-07:00 my-very-cool-host prometheus-podman-exporter[76891]: ts=2024-10-23T13:35:15.005Z caller=handler.go:105 level=info collector=system 2024-10-23T06:35:15.006149-07:00 my-very-cool-host prometheus-podman-exporter[76891]: ts=2024-10-23T13:35:15.005Z caller=handler.go:105 level=info collector=volume 2024-10-23T06:35:15.010302-07:00 my-very-cool-host prometheus-podman-exporter[76891]: panic: runtime error: invalid memory address or nil pointer dereference 2024-10-23T06:35:15.010409-07:00 my-very-cool-host prometheus-podman-exporter[76891]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xf85117] 2024-10-23T06:35:15.010470-07:00 my-very-cool-host prometheus-podman-exporter[76891]: goroutine 1 [running]: 2024-10-23T06:35:15.010517-07:00 my-very-cool-host prometheus-podman-exporter[76891]: github.com/containers/common/libimage.(Runtime).ListImages(0x0, {0x1bad728, 0xc0004157d0}, {0x0?, 0xc000533740?, 0x425ce5?}, 0x30?) 2024-10-23T06:35:15.010568-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/vendor/github.com/containers/common/libimage/runtime.go:584 +0x97 2024-10-23T06:35:15.010615-07:00 my-very-cool-host prometheus-podman-exporter[76891]: github.com/containers/podman/v4/pkg/domain/infra/abi.(ImageEngine).List(0xc0003363f0, {0x1bad728, 0xc0004157d0}, {0x0?, {0x0?, 0x0?, 0x0?}}) 2024-10-23T06:35:15.010776-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/vendor/github.com/containers/podman/v4/pkg/domain/infra/abi/images_list.go:25 +0x174 2024-10-23T06:35:15.010827-07:00 my-very-cool-host prometheus-podman-exporter[76891]: github.com/containers/prometheus-podman-exporter/pdcs.updateImages() 2024-10-23T06:35:15.010870-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/pdcs/image.go:48 +0x7c 2024-10-23T06:35:15.010923-07:00 my-very-cool-host prometheus-podman-exporter[76891]: github.com/containers/prometheus-podman-exporter/pdcs.StartEventStreamer({0x1ba0020?, 0xc00033a780}, 0x1) 2024-10-23T06:35:15.010965-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/pdcs/events.go:21 +0x1d3 2024-10-23T06:35:15.011004-07:00 my-very-cool-host prometheus-podman-exporter[76891]: github.com/containers/prometheus-podman-exporter/exporter.Start(0x0?, {0x0?, 0x0?, 0x0?}) 2024-10-23T06:35:15.011053-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/exporter/exporter.go:93 +0x5fb 2024-10-23T06:35:15.011093-07:00 my-very-cool-host prometheus-podman-exporter[76891]: github.com/containers/prometheus-podman-exporter/cmd.run(0x28be060?, {0xc0003546e0?, 0x4?, 0x18fafaf?}) 2024-10-23T06:35:15.011138-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/cmd/root.go:53 +0x1c 2024-10-23T06:35:15.011178-07:00 my-very-cool-host prometheus-podman-exporter[76891]: github.com/spf13/cobra.(Command).execute(0x28be060, {0xc0000402e0, 0x2, 0x2}) 2024-10-23T06:35:15.011216-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/vendor/github.com/spf13/cobra/command.go:987 +0xaa3 2024-10-23T06:35:15.011260-07:00 my-very-cool-host prometheus-podman-exporter[76891]: github.com/spf13/cobra.(Command).ExecuteC(0x28be060) 2024-10-23T06:35:15.011312-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/vendor/github.com/spf13/cobra/command.go:1115 +0x3ff 2024-10-23T06:35:15.011358-07:00 my-very-cool-host prometheus-podman-exporter[76891]: github.com/spf13/cobra.(*Command).Execute(...) 2024-10-23T06:35:15.011401-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/vendor/github.com/spf13/cobra/command.go:1039 2024-10-23T06:35:15.011443-07:00 my-very-cool-host prometheus-podman-exporter[76891]: github.com/containers/prometheus-podman-exporter/cmd.Execute() 2024-10-23T06:35:15.011485-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/cmd/root.go:61 +0x1e 2024-10-23T06:35:15.011524-07:00 my-very-cool-host prometheus-podman-exporter[76891]: main.main() 2024-10-23T06:35:15.011566-07:00 my-very-cool-host prometheus-podman-exporter[76891]: #011/opt/builder/BUILD/prometheus-podman-exporter-1.10.1/main.go:8 +0xf 2024-10-23T06:35:15.011647-07:00 my-very-cool-host systemd[23174]: prometheus-podman-exporter.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

podman system prune command fixes this.