panic in internal/weldr/api.go:205

[msehnout]

Reported output:

Oct 22 09:24:03 localhost.localdomain osbuild-composer[10896]: 2020/10/22 09:24:03 Loaded configuration:
Oct 22 09:24:03 localhost.localdomain osbuild-composer[10896]: [koji]
Oct 22 09:24:03 localhost.localdomain osbuild-composer[10896]:   ca = ""
Oct 22 09:24:03 localhost.localdomain osbuild-composer[10896]: [worker]
Oct 22 09:24:03 localhost.localdomain osbuild-composer[10896]:   ca = ""
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: 2020/10/22 09:24:06 http: panic serving @: runtime error: invalid memory address or nil pointer dereference
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: goroutine 9 [running]:
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: net/http.(*conn).serve.func1(0xc000384280)
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]:         /usr/lib/golang/src/net/http/server.go:1800 +0x13b
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: panic(0x55b9803956c0, 0x55b9807adcf0)
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]:         /usr/lib/golang/src/runtime/panic.go:975 +0x3e7
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: github.com/osbuild/osbuild-composer/internal/weldr.(*API).getComposeStatus(0xc0001ba900, 0xc0001bb000, 0x0, 0x55b98045ad80, 0xc0001bd680, 0xc000360000, 0x22cdb, 0x24000, 0xc000187b90, 0x2, ...)
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]:         /builddir/build/BUILD/osbuild-composer-22/_build/src/github.com/osbuild/osbuild-composer/internal/weldr/api.go:205 +0x230
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: github.com/osbuild/osbuild-composer/internal/weldr.(*API).composeQueueHandler(0xc0001ba900, 0x55b980454d00, 0xc00027e700, 0xc000394100, 0xc0001baf80, 0x1, 0x4)
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]:         /builddir/build/BUILD/osbuild-composer-22/_build/src/github.com/osbuild/osbuild-composer/internal/weldr/api.go:2016 +0x236
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc000180c60, 0x55b980454d00, 0xc00027e700, 0xc000394100)
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]:         /usr/share/gocode/src/github.com/julienschmidt/httprouter/router.go:387 +0xc39
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: github.com/osbuild/osbuild-composer/internal/weldr.(*API).ServeHTTP(0xc0001ba900, 0x55b980454d00, 0xc00027e700, 0xc000394100)
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]:         /builddir/build/BUILD/osbuild-composer-22/_build/src/github.com/osbuild/osbuild-composer/internal/weldr/api.go:161 +0x144
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: net/http.serverHandler.ServeHTTP(0xc00027e460, 0x55b980454d00, 0xc00027e700, 0xc000394100)
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]:         /usr/lib/golang/src/net/http/server.go:2836 +0xa5
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: net/http.(*conn).serve(0xc000384280, 0x55b9804563c0, 0xc00006e280)
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]:         /usr/lib/golang/src/net/http/server.go:1924 +0x86e
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]: created by net/http.(*Server).Serve
Oct 22 09:24:06 localhost.localdomain osbuild-composer[10896]:         /usr/lib/golang/src/net/http/server.go:2962 +0x35e
Oct 22 09:25:01 localhost.localdomain osbuild-composer[10947]: No match for group package "lsvpd"
Oct 22 09:25:01 localhost.localdomain osbuild-composer[10947]: No match for group package "powerpc-utils"
Oct 22 09:25:01 localhost.localdomain osbuild-composer[10947]: No match for group package "plymouth"
Oct 22 09:25:01 localhost.localdomain osbuild-composer[10947]: No match for group package "dracut-config-rescue"
Oct 22 09:25:01 localhost.localdomain osbuild-composer[10947]: No match for group package "firewalld"

How I triggered the issue:

1. I updated to the "testing" RPM in Fedora 32:

   ~ $ rpm -q osbuild-composer
   osbuild-composer-22-1.fc32.x86_64
   ~ $ rpm -V osbuild-composer
   ~ $

2. In cockpit I have an empty blueprint for testing, so I just created a "qcow2" image (which was already in cache because I created AWS image before the update to v22) and then I tried to display the queue which took a while to load so I checked the logs and here it comes.

Affected code: https://github.com/osbuild/osbuild-composer/blob/0e7a1dd4075d311e39de77959ea57f8bc5d7ccb0/internal/weldr/api.go#L205

[msehnout]

Worth mentioning: osbuild-composer "survived" the issue, the queue loaded eventually and the image has been created.

[larskarlitski]

Indeed, jobStatus can be nil when api.workers.JobStatus returns an error, which is ignored in the line above.

[ondrejbudai]

I tried reproducing this but failed. There must be some kind of race condition...

[msehnout]

I tried to write a script to trigger it, but failed as well. Only cockpit was able to do it for me.