panic: assignment to entry in nil map

[ianw]

Our CI jobs have started failing with

2022-04-22 05:45:36.243 | + podman run --name dib-tmp-export-28472 -d dib-tmp-work-image-13092 /bin/sh
2022-04-22 05:45:36.571 | panic: assignment to entry in nil map
2022-04-22 05:45:36.571 | 
2022-04-22 05:45:36.571 | goroutine 1 [running]:
2022-04-22 05:45:36.571 | github.com/opencontainers/runtime-tools/generate.(*Generator).addEnv(...)
2022-04-22 05:45:36.571 |   github.com/opencontainers/runtime-tools/generate/generate.go:525
2022-04-22 05:45:36.571 | github.com/opencontainers/runtime-tools/generate.(*Generator).AddProcessEnv(0xc000717bc8, {0x17edeb7, 0x8}, {0xc000351cc0, 0xc})
2022-04-22 05:45:36.571 |   github.com/opencontainers/runtime-tools/generate/generate.go:501 +0x2ea
2022-04-22 05:45:36.571 | github.com/containers/podman/libpod.(*Container).generateSpec(0xc00063cf20, {0x1ad3050, 0xc000124bd0})
2022-04-22 05:45:36.571 |   github.com/containers/podman/libpod/container_internal_linux.go:648 +0x2965
2022-04-22 05:45:36.571 | github.com/containers/podman/libpod.(*Container).init(0xc00063cf20, {0x1ad3050, 0xc000124bd0}, 0x0)
2022-04-22 05:45:36.571 |   github.com/containers/podman/libpod/container_internal.go:1098 +0x8e
2022-04-22 05:45:36.571 | github.com/containers/podman/libpod.(*Container).prepareToStart(0xc00063cf20, {0x1ad3050, 0xc000124bd0}, 0xa0?)
2022-04-22 05:45:36.571 |   github.com/containers/podman/libpod/container_internal.go:875 +0x345
2022-04-22 05:45:36.572 | github.com/containers/podman/libpod.(*Container).Start(0xc00063cf20, {0x1ad3050?, 0xc000124bd0?}, 0x0?)
2022-04-22 05:45:36.572 |   github.com/containers/podman/libpod/container_api.go:91 +0xde
2022-04-22 05:45:36.572 | github.com/containers/podman/pkg/domain/infra/abi.(*ContainerEngine).ContainerRun(0xc000518058, {0x1ad3050, 0xc000124bd0}, {{0x0, 0x0}, 0x1, {0x17f5ed4, 0xd}, 0xc000010020, 0x0, ...})
2022-04-22 05:45:36.572 |   github.com/containers/podman/pkg/domain/infra/abi/containers.go:947 +0x276
2022-04-22 05:45:36.572 | github.com/containers/podman/cmd/podman/containers.run(0x25046a0?, {0xc0004b4280?, 0x2, 0x5})
2022-04-22 05:45:36.572 |   github.com/containers/podman/cmd/podman/containers/run.go:194 +0x7e6
2022-04-22 05:45:36.572 | github.com/spf13/cobra.(*Command).execute(0x25046a0, {0xc00003c090, 0x5, 0x5})
2022-04-22 05:45:36.572 |   github.com/spf13/cobra/command.go:856 +0x67c
2022-04-22 05:45:36.572 | github.com/spf13/cobra.(*Command).ExecuteC(0x251b3a0)
2022-04-22 05:45:36.572 |   github.com/spf13/cobra/command.go:974 +0x3b4
2022-04-22 05:45:36.572 | github.com/spf13/cobra.(*Command).Execute(...)
2022-04-22 05:45:36.572 |   github.com/spf13/cobra/command.go:902
2022-04-22 05:45:36.572 | github.com/spf13/cobra.(*Command).ExecuteContext(...)
2022-04-22 05:45:36.572 |   github.com/spf13/cobra/command.go:895
2022-04-22 05:45:36.572 | main.Execute()
2022-04-22 05:45:36.572 |   github.com/containers/podman/cmd/podman/root.go:91 +0xc5
2022-04-22 05:45:36.572 | main.main()
2022-04-22 05:45:36.572 |   github.com/containers/podman/cmd/podman/main.go:39 +0x74

Several different types of builds have started failing with this. Concentrating on one type [1] the last successful build was [2] yesterday.

It looks like there has been a minor point-update of podman in Debian in that time, e.g. [3] (failing)

2022-04-22 06:44:54.031462 | ubuntu-focal | Get:69 http://deb.debian.org/debian unstable/main amd64 podman amd64 3.4.7+ds1-1 [9625 kB]

versus [4] (passing)

2022-04-21 20:08:20.771641 | ubuntu-focal | Get:69 http://deb.debian.org/debian unstable/main amd64 podman amd64 3.4.6+ds1-1 [9623 kB]

This seems to be about the only difference. It seems this is a release to fix a CVE [5]

* New upstream release.
     - Fixes:  [CVE-2022-1227](https://security-tracker.debian.org/tracker/CVE-2022-1227)

I haven't managed to track this down further yet, but it felt more like a generic issue than something Debian specific; we run from unstable because we need some other features so might be some early testers of this.

[1] https://zuul.opendev.org/t/openstack/builds?job_name=dib-nodepool-functional-openstack-fedora-35-containerfile-src&project=openstack/diskimage-builder [2] https://zuul.opendev.org/t/openstack/build/7fe19006ea0543afab58d56b206c70ed [3] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7de/838863/2/check/nodepool-build-image-siblings/7de1ad4/job-output.txt [4] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_806/836228/10/check/nodepool-build-image-siblings/806f881/job-output.txt [5] https://tracker.debian.org/news/1320098/accepted-libpod-347ds1-1-source-into-unstable/

[giuseppe]

it looks like the Debian package is using an old version of github.com/opencontainers/runtime-tools.

AddProcessEnv is not using addEnv since:

commit 3967c4654461e6673d2418e05678bcda4bf51b2f
Author: Giuseppe Scrivano <gscrivan@redhat.com>
Date:   Wed Aug 19 13:18:19 2020 +0200

    vendor: update opencontainers/runtime-spec

    Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

@siretart PTAL

[Luap99]

Sine this is is a debian issue you should report this to the distro specific bug tracker. Looks like the latest github.com/opencontainers/runtime-tools tag is from 2019, I guess there are using this instead.

[giuseppe]

@siretart sorry if it was already discussed in the past and I am not aware of the outcome but is there anything we can do to use the same version of the modules we ship and avoid such kind of issues?

[jqueuniet]

Had the same problem this morning with the last podman update for Debian sid. Upgrading to podman 4 from experimental solved my issue.

[siretart]

@giuseppe you can track the versions of containers/runtime-tools in Debian at https://tracker.debian.org/pkg/golang-github-opencontainers-runtime-tools. As you can see, it is packaging the lastest upstream release.

Any chance you can assist with having a new upstream release so that podman doesn't need to vendor unreleased versions? That would help me a lot. https://github.com/opencontainers/runtime-tools/issues/702 might be a good starting point, not sure where it got stuck.

Alternatively, you could help me by pointing out specific upstream commits that I could backport to the debian package (as you can see from 0.9.0+dfsg-4, I've been doing that e.g. for cd1349b7c47e9def74bee83f6cec88c5c3413e65 but evidently more changes are needed).

[Luap99]

see https://github.com/opencontainers/runtime-tools/issues/702

[giuseppe]

we can address the runtime-tools issue for now, but I am more worried about the fact the Go modules we use and test are different than the ones used on Debian. We could be more diligent and use only released tags, but we have no control over the indirect dependencies.

I somehow understand why it is done but it defeats the only advantage of Go binaries static linking.

[siretart]

it looks like the Debian package is using an old version of github.com/opencontainers/runtime-tools.

AddProcessEnv is not using addEnv [...]

@giuseppe I think it is a bit more complicated than that, I did backport that relevant change: https://sources.debian.org/src/golang-github-opencontainers-runtime-tools/0.9.0%2Bdfsg-4/debian/patches/ProcessEnvPerformance.patch/ -- so the addEnv function is used in the debian package.

Unfortunately, it appears to be not sufficient, and probably additional patches are necessary to avoid this crash.

[siretart]

Actually, I think the comment in https://github.com/containers/podman/blob/f65f3320e1124c94db053c1f811487920ae2a70e/libpod/container_internal_linux.go#L1529-L1531 is referring to exactly this crash.

I'm backporting upstream commit a42c131c80fc8c7220687c56cf4384a224572ca0 which reverts the usage to the deprecated API. Initial testing confirms that this avoids the crash. Go figure! (pun intended)