search

hashicorp / nomad-driver-podman

A nomad task driver plugin for sandboxing workloads in podman containers
https://developer.hashicorp.com/nomad/plugins/drivers/podman
Mozilla Public License 2.0
224 stars 61 forks source link

CI is failing with panics #301

Closed shoenig closed 7 months ago

shoenig commented 7 months ago

Noticed run-tests workflow is commonly failing with the panic below. I thought it might be the recent grpc update, but CI worked fine in that PR https://github.com/hashicorp/nomad-driver-podman/pull/294/checks

make: *** [GNUmakefile:86: test-ci] Error 1
2023-11-27T10:21:42.305Z [DEBUG] logmon/logmon.go:201: driver_harness.logmon: opening fifo: path=/tmp/nomad_driver_harness-1897751648/edcbe836-6230-0cbf-3faf-398b6d0fa3c2/alloc/logs/.tmpfs.stderr.fifo
2023-11-27T10:21:42.305Z [DEBUG] logmon/logmon.go:201: driver_harness.logmon: opening fifo: path=/tmp/nomad_driver_harness-3878092109/9448fbc9-ff90-a3d6-2294-207e84b7d89d/alloc/logs/.logJournald.stderr.fifo
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x49 pc=0x10d90cc]

goroutine 197 [running]:
github.com/hashicorp/nomad-driver-podman.(*Driver).StartTask(0xc0000e2000, 0xc0006482c0)
    /home/runner/work/nomad-driver-podman/nomad-driver-podman/driver.go:405 +0x6c
github.com/hashicorp/nomad/plugins/drivers.(*driverPluginServer).StartTask(0xc0000c6408, {0xc000787bc0?, 0x560f46?}, 0x0?)
    /home/runner/go/pkg/mod/github.com/hashicorp/nomad@v1.6.0/plugins/drivers/server.go:111 +0x58
github.com/hashicorp/nomad/plugins/drivers/proto._Driver_StartTask_Handler({0x1378dc0?, 0xc0000c6408}, {0x15f60a8, 0xc000787b90}, 0xc000200770, 0x0)
    /home/runner/go/pkg/mod/github.com/hashicorp/nomad@v1.6.0/plugins/drivers/proto/driver.pb.go:4502 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000f0000, {0x15fe780, 0xc0000e0b60}, 0xc0007c8d80, 0xc000317da0, 0x1e7a188, 0x0)
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.3/server.go:1335 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc0000f0000, {0x15fe780, 0xc0000e0b60}, 0xc0007c8d80, 0x0)
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.3/server.go:1712 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.3/server.go:947 +0xca
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.3/server.go:958 +0x15c
FAIL TestPodmanDriver_ReadOnlyFilesystem (-1.00s)
shoenig commented 7 months ago

Hmm weird, seems to be transient, e.g. just poking the CI to run and it worked https://github.com/hashicorp/nomad-driver-podman/actions/runs/7060967499/job/19221631557?pr=302