woodpecker-ci / autoscaler

Scale your woodpecker agents automatically to the moon and back based on the current load.
Apache License 2.0
29 stars 5 forks source link

Autoscaler segfaults on ARM64 when deploying on Hetzner #148

Closed golyalpha closed 1 month ago

golyalpha commented 2 months ago

I just deployed the autoscaler on my arm-based kubernetes cluster, which went fine, until it actually tried to deploy an agent on Hetzner.

Here's the log:

{"time":"2024-05-22T07:04:52Z","message":"Starting autoscaler with log-level=info"}
{"level":"info","agent":"pool-1-agent-rdBV","reason":"not found on provider","time":"2024-05-22T07:05:53Z","message":"remove agent"}
{"level":"info","agent":"pool-1-agent-TnVu","time":"2024-05-22T07:06:53Z","message":"deploying agent"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x434178]

goroutine 1 [running]:
go.woodpecker-ci.org/autoscaler/providers/hetznercloud.(*Provider).DeployAgent(0x4000196840, {0x643b20, 0xa1f800}, 0x4000248900)
    /src/providers/hetznercloud/provider.go:99 +0x198
go.woodpecker-ci.org/autoscaler/engine.(*Autoscaler).createAgents(0x400063fb40, {0x643b20, 0xa1f800}, 0x400063fa28?)
    /src/engine/autoscaler.go:85 +0x1a0
go.woodpecker-ci.org/autoscaler/engine.(*Autoscaler).Reconcile(0x400063fb40, {0x643b20, 0xa1f800})
    /src/engine/autoscaler.go:246 +0x118
main.run(0x4000185000)
    /src/cmd/woodpecker-autoscaler/main.go:85 +0x458
github.com/urfave/cli/v2.(*Command).Run(0x40001a26e0, 0x4000185000, {0x4000020170, 0x1, 0x1})
    /go/pkg/mod/github.com/urfave/cli/v2@v2.27.2/command.go:276 +0x71c
github.com/urfave/cli/v2.(*App).RunContext(0x4000194400, {0x643b20, 0xa1f800}, {0x4000020170, 0x1, 0x1})
    /go/pkg/mod/github.com/urfave/cli/v2@v2.27.2/app.go:333 +0x514
github.com/urfave/cli/v2.(*App).Run(...)
    /go/pkg/mod/github.com/urfave/cli/v2@v2.27.2/app.go:307
main.main()
    /src/cmd/woodpecker-autoscaler/main.go:123 +0x1a8
pat-s commented 1 month ago

AFAIK the autoscaler has already been successfully used on hetzner arm by @anbraten in the past?

anbraten commented 1 month ago

It seems to be coming from this line:

https://github.com/woodpecker-ci/autoscaler/blob/main/providers/hetznercloud/provider.go#L99

xoxys commented 1 month ago

@anbraten are you working on it already? Otherwise I would take it.

anbraten commented 1 month ago

Just had a quick look at it for now. Would be great if you could give it a try.

xoxys commented 1 month ago

The issue seems unrelated to the used architecture. @golyalpha what's the value of WOODPECKER_HETZNERCLOUD_SERVER_TYPE in your setup?

golyalpha commented 1 month ago

Pretty sure it was one of the ARM-based machines... Idk which one exactly anymore tho...