neondatabase / autoscaling

Postgres vertical autoscaling in k8s
Apache License 2.0
163 stars 21 forks source link

upgrade controller tools to 0.14 to avoid nil ptr deref #806

Closed skyzh closed 6 months ago

skyzh commented 8 months ago

I hit this issue on the dev machine. The controller tools panic when generating the files. Upgrading controller tools fix the issue.

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x101050bc0]
> go version
go version go1.22.0 linux/amd64

Not sure if this would impact production environment / is it necessary to do this upgrade now? Anyways, I'm just testing it out in my own environment so it's fine if we decide not to upgrade.

And I'm new to k8s so not sure if all these yaml files should be included in this pull request or not. Thanks for reviews!

cicdteam commented 8 months ago

is it ok that controller-tools v0.14.0 depends on k8s 1.29 but autoscaling requires 1.25?

from controller-tool go.mod

    k8s.io/api v0.29.0
    k8s.io/apiextensions-apiserver v0.29.0
    k8s.io/apimachinery v0.29.0
skyzh commented 8 months ago

maybe not..? and probably that's why i cannot spin up any VM for the past hour 🤣 maybe i'll set up a user-local go compiler on devserver instead.

sharnoff commented 8 months ago

In general, the issue we have is that the scheduler must have a version equal or -1 from the k8s API server, otherwise it'll just silently not work. And because everything has the same go.mod, every component will in practice use the same k8s version as the scheduler. Our local dev, staging, and prod clusters are all 1.25, so while in theory it's possible for us to use k8s 1.25 deps with code generated by controller-tools for 1.29, IMO it's likely to have hard-to-debug errors.

skyzh commented 8 months ago

Let's keep this pull request on hold and don't merge. TL;DR, anyone working on autoscaling will need to use go <= 1.21.

Omrigan commented 6 months ago

Blocked on k8s upgrade