search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.39k stars 932 forks source link

`simplestream-maintainer` errors out with `panic: assignment to entry in nil map` #13451

Closed simondeziel closed 6 months ago

simondeziel commented 6 months ago

Since May 5th, simplestream-maintainer panics prevent publication of new images to images:. Here's the logs from the first panic:

May 05 01:02:08 images systemd[1]: Starting Simplestream maintainer...
May 05 01:02:13 images simplestream-maintainer[220716]: time=2024-05-05T01:02:13.155Z level=INFO msg="New version added to the product catalog" streamName=images product=funtoo:next:amd64:default version=20240505_0018
May 05 01:02:15 images simplestream-maintainer[220716]: time=2024-05-05T01:02:15.745Z level=INFO msg="New version added to the product catalog" streamName=images product=centos:8-Stream:amd64:default version=20240505_0021
May 05 01:02:16 images simplestream-maintainer[220716]: time=2024-05-05T01:02:16.498Z level=INFO msg="New version added to the product catalog" streamName=images product=alpine:3.19:amd64:cloud version=20240505_0018
May 05 01:02:16 images simplestream-maintainer[220716]: time=2024-05-05T01:02:16.630Z level=INFO msg="New version added to the product catalog" streamName=images product=fedora:40:amd64:cloud version=20240505_0004
May 05 01:02:16 images simplestream-maintainer[220716]: time=2024-05-05T01:02:16.903Z level=INFO msg="New version added to the product catalog" streamName=images product=devuan:chimaera:amd64:default version=20240505_0009
May 05 01:02:20 images simplestream-maintainer[220716]: time=2024-05-05T01:02:20.720Z level=INFO msg="New version added to the product catalog" streamName=images product=almalinux:9:amd64:default version=20240505_0011
May 05 01:02:21 images simplestream-maintainer[220716]: time=2024-05-05T01:02:21.041Z level=INFO msg="New version added to the product catalog" streamName=images product=fedora:38:amd64:cloud version=20240505_0004
May 05 01:02:21 images simplestream-maintainer[220716]: time=2024-05-05T01:02:21.260Z level=INFO msg="New version added to the product catalog" streamName=images product=devuan:beowulf:amd64:default version=20240505_0009
May 05 01:02:22 images simplestream-maintainer[220716]: time=2024-05-05T01:02:22.903Z level=INFO msg="New version added to the product catalog" streamName=images product=oracle:9:amd64:default version=20240505_0011
May 05 01:02:24 images simplestream-maintainer[220716]: time=2024-05-05T01:02:24.256Z level=INFO msg="New version added to the product catalog" streamName=images product=fedora:38:amd64:default version=20240505_0004
May 05 01:02:24 images simplestream-maintainer[220716]: time=2024-05-05T01:02:24.304Z level=INFO msg="New version added to the product catalog" streamName=images product=alpine:3.16:amd64:cloud version=20240505_0017
May 05 01:02:27 images simplestream-maintainer[220716]: time=2024-05-05T01:02:27.700Z level=INFO msg="New version added to the product catalog" streamName=images product=debian:bookworm:amd64:cloud version=20240505_0002
May 05 01:02:28 images simplestream-maintainer[220716]: time=2024-05-05T01:02:28.692Z level=INFO msg="New version added to the product catalog" streamName=images product=centos:9-Stream:amd64:default version=20240505_0021
May 05 01:02:29 images simplestream-maintainer[220716]: time=2024-05-05T01:02:29.965Z level=INFO msg="New version added to the product catalog" streamName=images product=mint:virginia:amd64:cloud version=20240505_0004
May 05 01:02:30 images simplestream-maintainer[220716]: time=2024-05-05T01:02:30.356Z level=INFO msg="New version added to the product catalog" streamName=images product=fedora:39:amd64:cloud version=20240505_0004
May 05 01:02:30 images simplestream-maintainer[220716]: time=2024-05-05T01:02:30.749Z level=INFO msg="New version added to the product catalog" streamName=images product=busybox:1.36.1:amd64:default version=20240505_0008
May 05 01:02:33 images simplestream-maintainer[220716]: time=2024-05-05T01:02:33.676Z level=INFO msg="New version added to the product catalog" streamName=images product=alt:p10:amd64:default version=20240505_0008
May 05 01:02:34 images simplestream-maintainer[220716]: time=2024-05-05T01:02:34.824Z level=INFO msg="New version added to the product catalog" streamName=images product=devuan:chimaera:amd64:cloud version=20240505_0009
May 05 01:02:35 images simplestream-maintainer[220716]: time=2024-05-05T01:02:35.326Z level=INFO msg="New version added to the product catalog" streamName=images product=oracle:7:amd64:cloud version=20240505_0011
May 05 01:02:36 images simplestream-maintainer[220716]: time=2024-05-05T01:02:36.027Z level=INFO msg="New version added to the product catalog" streamName=images product=mint:una:amd64:default version=20240505_0004
May 05 01:02:36 images simplestream-maintainer[220716]: time=2024-05-05T01:02:36.387Z level=INFO msg="New version added to the product catalog" streamName=images product=gentoo:current:amd64:openrc version=20240505_0008
May 05 01:02:36 images simplestream-maintainer[220716]: time=2024-05-05T01:02:36.715Z level=INFO msg="New version added to the product catalog" streamName=images product=mint:una:amd64:cloud version=20240505_0004
May 05 01:02:37 images simplestream-maintainer[220716]: time=2024-05-05T01:02:37.365Z level=INFO msg="New version added to the product catalog" streamName=images product=alpine:3.18:amd64:cloud version=20240505_0018
May 05 01:02:37 images simplestream-maintainer[220716]: panic: assignment to entry in nil map
May 05 01:02:37 images simplestream-maintainer[220716]: goroutine 34 [running]:
May 05 01:02:37 images simplestream-maintainer[220716]: main.buildProductCatalog.func2()
May 05 01:02:37 images simplestream-maintainer[220716]:         github.com/canonical/lxd-imagebuilder/simplestream-maintainer/cmd_build.go:305 +0x42f
May 05 01:02:37 images simplestream-maintainer[220716]: main.buildProductCatalog.func1()
May 05 01:02:37 images simplestream-maintainer[220716]:         github.com/canonical/lxd-imagebuilder/simplestream-maintainer/cmd_build.go:238 +0x39
May 05 01:02:37 images simplestream-maintainer[220716]: created by main.buildProductCatalog in goroutine 1
May 05 01:02:37 images simplestream-maintainer[220716]:         github.com/canonical/lxd-imagebuilder/simplestream-maintainer/cmd_build.go:228 +0x44b
May 05 01:02:37 images systemd[1]: simplestream-maintainer.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 05 01:02:37 images systemd[1]: simplestream-maintainer.service: Failed with result 'exit-code'.
May 05 01:02:37 images systemd[1]: Failed to start Simplestream maintainer.
May 05 01:02:37 images systemd[1]: simplestream-maintainer.service: Consumed 35.370s CPU time.

Note, this simplestream-maintainer binary is not from the newly created lxd-imagebuilder snap so maybe this panic would be fixed by switching to the snap (PR not yet merged).

tomponline commented 6 months ago

@simondeziel impact of this? Images getting out of date?

Sounds like a high priority @MusicDin

MusicDin commented 6 months ago

The impact of this issue is quite critical as it prevents rebuilding the product catalog.

Products are never added to the catalog if there is no version (image) associated with it. However, when a certain product no longer contains any version, the empty product remained in the catalog after pruning (for example, we no longer build Funtoo 1.4, so it does not contain any versions and should be removed from the catalog). This caused a panic because field versions is nil for a product from the catalog.