Compiling downloads and uses "invalid version" of packages (unavailable from package repo)

Problem

Building software with removed versions can increase risk of build failures, runtime bugs, and downtime. Package maintainers sometimes remove a version from their official repository (e.g. when problems justify removing it).

Our builds (directly or indirectly in go.mod) seem to rely on cached "invalid version" of packages that are unavailable from package's official repository. For example, this "invalid version" looks like a prerelease from May 11, 2020:

https://github.com/onflow/flow-go/blob/23c304d4c038206cb0d4857b4dfa668bfc4d782b/go.mod#L28

Valid versions of go-grpc-middleware include v2.0.1 (Sep 2023) and v2.0.0-rc.2 (June 2020).

If "invalid version" errors were detected and those versions were already re-evaluated for continued use, then maybe we can provide docs to address potential concerns.

Bypassing vs Detecting "Invalid Version" Errors

Not all go distributions report "invalid version: unknown revision" errors with default GOPROXY setting:

golang.org uses GOPROXY=https://proxy.golang.org,direct
Red Hat (Fedora Silverblue, etc.) uses GOPROXY=direct

Using GOPROXY=https://proxy.golang.org,direct bypasses "invalid version" error and downloads cached copy of invalid version from the specified proxy.

One way to detect "invalid version" errors is to use GOPROXY=direct.

Examples of Detected "Invalid Version" Error

We can specify GOPROXY=direct with commands like go build, go get, and go mod tidy. This is able to detect "invalid version" in our own go.mod as well as in go.mod of packages used.

$ go clean -modcache
$ GOPROXY="direct" go mod tidy
...
go: github.com/grpc-ecosystem/go-grpc-middleware/providers/zerolog/v2@v2.0.0-rc.2 requires
    github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-20200501113911-9a95f0fdbfea: invalid version: unknown revision 9a95f0fdbfea

First "invalid version" error reported by go is a go-grpc-middleware prerelease of v2.0.0 (May 11, 2020). Examples of valid versions include v2.0.1 (Sep 21, 2023) and v2.0.0-rc.2 (June 19, 2020).

If we edit go.mod to suppress the first "invalid version" error, then the next error is reported about a different package.

$ vim go.mod
$ GOPROXY="direct" go mod tidy
...
go: github.com/ipfs/go-ipfs-provider@v0.7.0 requires
    github.com/ipfs/go-ipfs-blockstore@v0.2.0: reading github.com/ipfs/go-ipfs-blockstore/go.mod at revision v0.2.0: unknown revision v0.2.0

The "unknown revision" error is for v0.2.0 of go-ipfs-blockstore which isn't listed at https://github.com/ipfs/go-ipfs-blockstore/tags. Examples of valid versions include v1.3.1 (May 23, 2023) and v0.2.1 (Nov 17, 2021).

Caveats

Priority was not assigned because it depends on several factors (technical impact, non-technical impact, availability of appropriate resources, etc.) and it can be time-consuming to determine which fixed bugs exist in old invalid versions.

This issue was discovered while trying to compile cmd/util with atree storage migration PR #4633 for the 1st time before updating it to test Cadence PR 2882.

onflow / flow-go