caa: fixing toolchain not available

[beraldoleal]

make here is returning the following error:

go: download go1.22 for linux/amd64: toolchain not available

This change specifies the exact toolchain version (1.22.7) instead of the language family version (1.22), a feature introduced in Go 1.21. It ensures more precise toolchain resolution, improves reproducibility, and avoids potential automatic upgrades to future versions.

Updating go.mod to use 1.22.7 instead.

[wainersm]

Hi @beraldoleal !

I'm not sure I understood the problem. What's exactly the command you trying that resulted in

go: download go1.22 for linux/amd64: toolchain not available

?

Also, what's the go version in your env?

This change broke some static checkers like lint / golangci-lint and lint / go mod tidy. Apparently if you set the go version (instead of family version) then you need to set the exactly toolchain version.

[beraldoleal]

Hi @beraldoleal !

I'm not sure I understood the problem. What's exactly the command you trying that resulted in

go: download go1.22 for linux/amd64: toolchain not available

?

Just a make as defined at DEVELOPMENT.md file.

Also, what's the go version in your env?

1.22 as it is described at go.mod.

This change broke some static checkers like lint / golangci-lint and lint / go mod tidy. Apparently if you set the go version (instead of family version) then you need to set the exactly toolchain version.

What is the go version those linters are using? I'm not a go expert, but my understand is that we can use both options:

module <module>

go 1.22.N

Or

module <module>

go 1.22
toolchain go1.22.N

[beraldoleal]

I believe the main issue is that, previously, we could specify a "Go language version" in go.mod using just the major version (e.g., go 1.X). However, starting with Go 1.21+, this is no longer allowed. According to the documentation:

A go directive indicates that a module was written assuming the semantics of a given version of Go. The version must be a valid Go version, such as 1.9, 1.14, or 1.21rc1. The go directive sets the minimum version of Go required to use this module. Before Go 1.21, the directive was advisory only; now it is a mandatory requirement: Go toolchains refuse to use modules declaring newer Go versions.

So, since Go 1.22 is not yet a valid release, it cannot be used.

[stevenhorsman]

What is the go version those linters are using?

We use go 1.22.7 as per our versions.yaml

so it seems like:

go 1.22
toolchain go1.22.7

might help then, though it would add another thing to manually manage on go updates which is a pain.

I'm also not seeing this error in our CI, or locally, when I do make but maybe that is as I have 1.22.7 installed?

[beraldoleal]

What is the go version those linters are using?

We use go 1.22.7 as per our versions.yaml

so it seems like:

go 1.22
toolchain go1.22.7

might help then, though it would add another thing to manually manage on go updates which is a pain.

I agree, that is the reason I choose to modify only the go directive. I will update to use 1.22.7.

I'm also not seeing this error in our CI, or locally, when I do make but maybe that is as I have 1.22.7 installed?

This could be related to how your system retrieves cached toolchain versions. If the system doesn't have a toolchain installed that satisfy the requirement, it tries to derive and download the toolchain from go 1.22 in go.mod, but since Go 1.22 isn't released, it fails.

[wainersm]

Hi @beraldoleal ,

I wasn't understanding the problem, then I decided to give it a try with your changes (see below). I didn't know go was able to download/upgrade the system's installation; I always relied on packaged go (from Fedora) where If I hadn't the mininum version required I would either upgrade the package or manually download a version (to ~/.local). I confess that I dislike that idea of go upgrading itself auto-magically as I always preferred to control the version on my system myself.

Some questions that I have:

• Do we want that feature enabled by default?
• What if I have a newer than required (say 1.23.1), will it download to match exactly 1.22.7?
• I'm desperate to roll back the changes it did to my system. do you know how? :)

$ export PATH=~/.local/lib/go1.21.13/bin:$PATH
$ go version
go version go1.21.13 linux/amd64
$ cd src/cloud-api-adaptor/
$ make
go: downloading go1.22.0 (linux/amd64)
GOOS=linux GOARCH=amd64 CGO_ENABLED=1 go build -ldflags="-X 'github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/cmd.VERSION=v0.9.0-alpha.3-dev' -X 'github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/cmd.COMMIT=711ee01f1e37811f3f3e6d3033b73996a22cf33f'" -tags=aws,azure,gcp,ibmcloud,vsphere,libvirt,docker -o "cloud-api-adaptor" ./cmd/cloud-api-adaptor
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-X 'github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/cmd.VERSION=v0.9.0-alpha.3-dev' -X 'github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/cmd.COMMIT=711ee01f1e37811f3f3e6d3033b73996a22cf33f'" -tags=aws,azure,gcp,ibmcloud,vsphere,libvirt,docker -o "agent-protocol-forwarder" ./cmd/agent-protocol-forwarder
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-X 'github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/cmd.VERSION=v0.9.0-alpha.3-dev' -X 'github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/cmd.COMMIT=711ee01f1e37811f3f3e6d3033b73996a22cf33f'" -tags=aws,azure,gcp,ibmcloud,vsphere,libvirt,docker -o "process-user-data" ./cmd/process-user-data
$ go version
go version go1.22.0 linux/amd64
$ which go
~/.local/lib/go1.21.13/bin/go

[beraldoleal]

Hi @beraldoleal ,

I wasn't understanding the problem, then I decided to give it a try with your changes (see below). I didn't know go was able to download/upgrade the system's installation; I always relied on packaged go (from Fedora) where If I hadn't the mininum version required I would either upgrade the package or manually download a version (to ~/.local). I confess that I dislike that idea of go upgrading itself auto-magically as I always preferred to control the version on my system myself.

TBH, I'm having mix feelings with this new feature as well, as I was used to gvm. But I'm trying to embrace this new mechanism as a good thing and I think that makes sense, since it will make the development life much easier.

Not a go expert here, but let me try to address your question...

Some questions that I have:

* Do we want that feature enabled by default?

This controlled by the GOTOOLCHAIN env variable, which by default is set to "auto". So, if the system has Go 1.21+, the default settings and a gotoolset that does not satisfy the requirement, a simple make will trigger this new logic to download the toolchain as defined in our project, which will fail. Previously, it acted as a suggestion for the language version. Starting with 1.21, its now specifying an exact toolchain release instead of just the language version.

If you want to disable this, each developer would need to set GOTOOLCHAIN=local to avoid downloading anything.

I'm not sure if adding this to our Makefile is the right choice, tbh. This would take us in a different direction from what Go community has decided (and lots of go projects are adopting including k8s). Pre-installing Go SDKs is a maintenance burden for devs, CIs, and build systems. It's hard for them to keep up to date with the latest Go or with the pace K8S adopts them (like within a week of Go's upstream release).

A test that you can do is trying to run the same experiment from the main branch, but with a go 1.21 installed. This should trigger the download mechanism that will fail.

* What if I have a newer than required (say 1.23.1), will it download to match exactly 1.22.7?

The go directive specifies the minimum required version. If you have the toolchain that satisfy this requirement, my understanding is that will not download.

* I'm desperate to roll back the changes it did to my system. do you know how? :)

lol.

Now, with this new logic, the toolset is just like another module, fetched globally and shared across projects. If you are outside the project folder your go version should give you the local go version. But if you would like to enforce that while still under the project folder, you need to run export GOTOOLCHAIN=local.

Let me know if that makes sense to you.

[beraldoleal]

@wainersm @stevenhorsman any comments?

[stevenhorsman]

So I guess my concern is the benefit we get versus the extra bumps we need to do whenever we bump go versions.

This would take us in a different direction from what Go community has decided (and lots of go projects are adopting including k8s).

My question about this is that K8s seemed to only need to bump for minor versions, not fix versions (they when from 1.22.0 -> 1.23.0 with 7 months gap in between). This level of manual bumping I'm fine with, but that seemed to cause issues with our linters?

[beraldoleal]

@stevenhorsman has discussed during the meeting. here is with 1.22.0 passing CI.