Tar errors on cache restore after toolchain installation

matthewhughes-uw commented 1 year ago

Description:

When the version of Go in the go directive of go.mod is different from the version in the toolchain directive there is a cache conflict which causes errors on cache extraction.

Note: I think this is related to, but different from https://github.com/actions/setup-go/issues/403 since that appears to be covering the situation where some other tool is also working on the cache, but this issue is purely for the action running on a hosted runner without any additional changes to the environment etc.

Action version:

actions/setup-go@v4

Platform:

[x] Ubuntu
[ ] macOS (untested)
[ ] Windows (untested)

Runner type:

[x] Hosted
[ ] Self-hosted

Tools version:

go 1.21.0
toolchain go1.21.1

Repro steps:

See repo https://github.com/matthewhughes-uw/setup-go-test/, very basic repo with a single workflow that just runs this action (https://github.com/matthewhughes-uw/setup-go-test/blob/main/.github/workflows/go.yaml):

First run of the action https://github.com/matthewhughes-uw/setup-go-test/actions/runs/6195831453/job/16821285537

Installs Go 1.21.0 (version detected from go-version-file
Go detects toolchain and performs a toolchain install of Go 1.21.1 (see output go: downloading go1.21.1 (linux/amd64))
This places some files under ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.1.linux-amd64
Finds no existing cache, doesn't try to extract anything
Caches /home/runner/go/pkg/mod (which contains the files above) and /home/runner/.cache/go-build

Second run of the action https://github.com/matthewhughes-uw/setup-go-test/actions/runs/6195844389/job/16821321968

First two steps as above
Finds an existing cache, attempts to extract it, but because of the toolchain installation step files under ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.1.linux-amd64 are already present on the machine, causing errors from tar

Expected behavior:

Caching of the toolchain is handled gracefully, no errors between runs that don't contain any extra caching logic outside of what the action provides by default

Actual behavior:

Errors on cache extraction (see repro steps)

dsame commented 1 year ago

Hello @matthewhughes-uw , thank you for the input. We are starting to investigate the problem.

thall commented 8 months ago

Hello @matthewhughes-uw , thank you for the input. We are starting to investigate the problem.

do you have any updates around this to share?

HaraldNordgren commented 8 months ago

@matthewhughes-uw Any updates on this? After updating to Go version:

--- a/go.mod
+++ b/go.mod
@@ -1,6 +1,8 @@
 module github.com/company/repo

-go 1.20
+go 1.21.1
+
+toolchain go1.21.6

 require (

We start having these warnings in the build:

/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/mkzip.go: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/zoneinfo.zip: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/README: Cannot open: File exists
...

HaraldNordgren commented 8 months ago

The issue seems to be caused by the setup-go being confused about the difference in "Go" version and the newly introduced "Toolchain" version.

It first downloads and caches 1.21.1
Then tries to download 1.21.6 as toolchain version
But for some reason this 1.21.6 download defaults back to 1.21.1, which it tries to download and cache
And boom... it 1.21.1 already exists in the cache from the first step! 🧨

15s
Run actions/setup-go@v4
  with:
    go-version-file: go.mod
    check-latest: true
    token: ***
    cache: true
Setup go version spec [1](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:1).[2](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:2)1.1
Attempting to resolve the latest version from the manifest...
matching 1.21.1...
Resolved as '1.21.1'
Attempting to download 1.21.1...
matching 1.21.1...
Acquiring 1.21.1 from https://github.com/actions/go-versions/releases/download/1.21.1-6107260229/go-1.21.1-linux-x64.tar.gz
Extracting Go...
/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/_temp/0b61aba1-9a59-4690-b4c0-4f21d65b8487 -f /home/runner/work/_temp/b0a91240-cd7a-42cc-bc74-0757c51b5e9b
Successfully extracted go to /home/runner/work/_temp/0b61aba1-9a59-4690-b4c0-4f21d65b8487
Adding to the cache ...
Successfully cached go to /opt/hostedtoolcache/go/1.21.1/x64
Added go to the path
Successfully set up Go version 1.21.1
go: downloading go1.21.6 (linux/amd64)
/opt/hostedtoolcache/go/1.21.1/x64/bin/go env GOMODCACHE
/opt/hostedtoolcache/go/1.21.1/x64/bin/go env GOCACHE
/home/runner/go/pkg/mod
/home/runner/.cache/go-build
Received 151688858 of 151688858 (100.0%), 166.5 MBs/sec
Cache Size: ~145 MB (151688858 B)
/usr/bin/tar -xf /home/runner/work/_temp/22489b85-a7d8-49[3](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:3)7-8083-e821[4](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:4)04[5](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:5)3fc7/cache.tzst -P -C /home/runner/work/hive/hive --use-compress-program unzstd
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.[6](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:6).linux-amd64/lib/time/mkzip.go: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.[21](https://github.com/dietdoctor/hive/actions/runs/7553469921/job/20564369667#step:3:22).6.linux-amd64/lib/time/zoneinfo.zip: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/README: Cannot open: File exists
/usr/bin/tar: ../../../go/pkg/mod/golang.org/toolchain@v0.0.1-go1.21.6.linux-amd64/lib/time/update.bash: Cannot open: File exists
...

matthewhughes-uw commented 8 months ago

The issue seems to be caused by the setup-go being confused about the difference in "Go" version and the newly introduced "Toolchain" version.
* It first downloads and caches 1.21.1
* Then tries to download 1.21.6 as toolchain version
* But for some reason this 1.21.6 download defaults back to 1.21.1, which it tries to download and cache
* And boom... it 1.21.1 already exists in the cache from the first step! 🧨

Note: the toolchain download is not done by setup-go, it's done by Go itself, e.g.

module example.com/my-mod

go 1.21.0

toolchain go1.21.1

Then if you try to run a go command using a version that doesn't match the toolchain the toolchain will be downloaded (creating new caches to ensure we download the new toolchain, and not just silently use an existing one):

$ mkdir gocache gomodcache
$ GOCACHE=$PWD/gocache GOMODCACHE=$PWD/gomodcache/ go1.21.0 env
go: downloading go1.21.1 (linux/amd64)

So unless setup-go is configured to use the go version (or perhaps, any newer version) specified in the toolchain directive then there will be this download. setup-go then continues to use the version it was configured with to run any future go commands (which may just invoke the toolchain version in the background)

$ GOCACHE=$PWD/gocache GOMODCACHE=$PWD/gomodcache/ go1.21.0 version
go version go1.21.1 linux/amd64

EDIT: I'm not sure if setup-go could just set the GOTOOLCHAIN env var to ensure it always use the configured version (docs https://go.dev/doc/toolchain)

The GOTOOLCHAIN environment setting can force a specific Go version, overriding the go and toolchain lines (and possibly just use the version from the toolchain directive by default)

$ GOTOOLCHAIN=go1.21.0 GOCACHE=$PWD/gocache GOMODCACHE=$PWD/gomodcache/ go1.21.0 version 
go version go1.21.0 linux/amd64

(from a quick test this skipped any toolchain download)

HaraldNordgren commented 8 months ago

@matthewhughes-uw That makes sense. What I don't understand is why Go on the second download defaults back from downloading 1.21.6 to instead getting 1.21.1 again.

Because if this didn't happen, then there would be no cache problem.

How I solved it for now is to upgrade using go get go, upgrading to 1.21.6 locally, which then removes the toolchain line altogether, since Go and Toolchain versions are now the same.

Then the caching issue does not appear.

matthewhughes-uw commented 8 months ago

What I don't understand is why Go on the second download defaults back from downloading 1.21.6 to instead getting 1.21.1 again.

There's a mix of Go and setup-go downloading bits. What I understand from your previous point:

It first downloads and caches 1.21.1: This is setup-go, and ends with the line Successfully set up Go version 1.21.1
Then tries to download 1.21.6 as toolchain version: setup-go then invokes go (specifically /opt/hostedtoolcache/go/1.21.1/x64/bin/go) which triggers a download of the toolchain, this is the line go: downloading go1.21.6 (linux/amd64)
But for some reason this 1.21.6 download defaults back to 1.21.1, which it tries to download and cache for go1.21.1: This is setup-go downloading the requested cache (not go doing anything)
And boom... it 1.21.1 already exists in the cache from the first step! 🧨: the issue is toolchain@v0.0.1-go1.21.6.linux-amd64/ exists in the cache (the toolchain downloaded from the 2nd step)

So the issue is we populate some of go1.21.1's GOMODCACHE and then restore the cache, which then complains because there are already some files there

You could remove the toolchain directive from your go.mod (unless you do have a requirement for a specific version of Go greater than what the go directive specifies when working in the module, see https://go.dev/blog/toolchain#toolchain-management)

spencerschrock commented 7 months ago

One way of avoiding this could be if actions/setup-go had a way of reading the toolchain directive instead of the go directive. Although it's still useful to test against the minimum version specified by the go directive, so I don't think changing the behavior of go-version-file is a good solution.

You could do your own toolchain version parsing in an earlier step, and set an env var or something:

uses: actions/setup-go@v5
with:
    go-version: ${{ env.GO_VERSION }}

matthewhughes-uw commented 7 months ago

I think this can be avoided by just telling Go to use the local toolchain when running go version before downloading the cache, see: https://github.com/actions/setup-go/pull/456

I think there's another can of worms/separate issue around which version of Go should this action use when a toolchain directive is present, e.g. at the moment if you specify go-version but have a go.mod specifying a newer-version of Go I expect any go command will run using the toolchain version, which may be unexpected

lukeed commented 5 months ago

For those finding this issue, the current workaround is to just ensure that your with/go-version value exactly matches the version specified in go.mod

// go.mod
go 1.22.2

# workflows/ci.yml
- uses: actions/setup-go@v5
  with:
    go-version: 1.22.2

matthewhughes-uw commented 5 months ago

For those finding this issue, the current workaround is to just ensure that your with/go-version value exactly matches the version specified in go.mod

You could also set the GOTOOLCHAIN env var to local:

    - name: Configure env for toolchain ignore
      shell: bash
      run: echo "GOTOOLCHAIN=local" >> "$GITHUB_ENV"
    - name: Setup Go
      uses: actions/setup-go@v5

see https://github.com/actions/setup-go/pull/460 for more details

timandy commented 1 month ago

The same issue for me. Only go1.23 has the problem.

https://github.com/timandy/gohack/actions/runs/10399737890

I found that disable cache can solve the problem.

      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
          cache: false

actions / setup-go

Tar errors on cache restore after toolchain installation #424