[bug] "Generate builder" and "Run sigstore/cosign-installer" steps failing with `error updating to TUF remote mirror: invalid key`

[jkreileder]

The "Generate builder" and "Run sigstore/cosign-installer" steps have started failing for my workflows. This used to work fine, not sure if it is just an intermittent error or something more fundamental:

Here's a build that worked 18 hours ago but is failing now (i.e. without any code changes): https://github.com/jkreileder/cf-ips-to-hcloud-fw/actions/runs/8339143012 (corresponding workflow)

1. Generate builder error:

Verifying artifact slsa-generator-container-linux-amd64: FAILED: error retrieving Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
    "mirror": "https://tuf-repo-cdn.sigstore.dev/",
    "metadata": {
        "root.json": {
            "version": 9,
            "len": 6766,
            "expiration": "12 Sep 24 06:53 UTC",
            "error": ""
        },
        "snapshot.json": {
            "version": 132,
            "len": 2302,
            "expiration": "09 Apr 24 16:16 UTC",
            "error": ""
        },
        "targets.json": {
            "version": 9,
            "len": 5478,
            "expiration": "12 Sep 24 06:13 UTC",
            "error": ""
        },
        "timestamp.json": {
            "version": 169,
            "len": 723,
            "expiration": "26 Mar 24 16:16 UTC",
            "error": ""
        }
    }
}

FAILED: SLSA verification failed: error retrieving Rekor public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
    "mirror": "https://tuf-repo-cdn.sigstore.dev/",
    "metadata": {
        "root.json": {
            "version": 9,
            "len": 6766,
            "expiration": "12 Sep 24 06:53 UTC",
            "error": ""
        },
        "snapshot.json": {
            "version": 132,
            "len": 2302,
            "expiration": "09 Apr 24 16:16 UTC",
            "error": ""
        },
        "targets.json": {
            "version": 9,
            "len": 5478,
            "expiration": "12 Sep 24 06:13 UTC",
            "error": ""
        },
        "timestamp.json": {
            "version": 169,
            "len": 723,
            "expiration": "26 Mar 24 16:16 UTC",
            "error": ""
        }
    }
}
Error: Process completed with exit code 6.

2. Run sigstore/cosign-installer@6e04d228eb30da1757ee4e1dd75a0ec73a653e06 error:

Error: getting ctlog public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
    "mirror": "https://tuf-repo-cdn.sigstore.dev/",
    "metadata": {
        "root.json": {
            "version": 9,
            "len": 6766,
            "expiration": "12 Sep 24 06:53 UTC",
            "error": ""
        },
        "snapshot.json": {
            "version": 132,
            "len": 2302,
            "expiration": "09 Apr 24 16:16 UTC",
            "error": ""
        },
        "targets.json": {
            "version": 9,
            "len": 5478,
            "expiration": "12 Sep 24 06:13 UTC",
            "error": ""
        },
        "timestamp.json": {
            "version": 169,
            "len": 723,
            "expiration": "26 Mar 24 16:16 UTC",
            "error": ""
        }
    }
}
main.go:74: error during command execution: getting ctlog public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
    "mirror": "https://tuf-repo-cdn.sigstore.dev/",
    "metadata": {
        "root.json": {
            "version": 9,
            "len": 6766,
            "expiration": "12 Sep 24 06:53 UTC",
            "error": ""
        },
        "snapshot.json": {
            "version": 132,
            "len": 2302,
            "expiration": "09 Apr 24 16:16 UTC",
            "error": ""
        },
        "targets.json": {
            "version": 9,
            "len": 5478,
            "expiration": "12 Sep 24 06:13 UTC",
            "error": ""
        },
        "timestamp.json": {
            "version": 169,
            "len": 723,
            "expiration": "26 Mar 24 16:16 UTC",
            "error": ""
        }
    }
}
Error: Process completed with exit code 1.

[tirumerla]

Started seeing the same just now.

[brad-getpassport]

+1 Seeing this across all our builds in the last 2 hours... this is holding up our release candidate...

Generating ephemeral keys...
Retrieving signed certificate...
creating signer: getting signer: getting key from Fulcio: getting CTFE public keys: updating local metadata and targets: error updating to TUF remote mirror: invalid key
remote status:{
    "mirror": "https://tuf-repo-cdn.sigstore.dev/",
    "metadata": {
        "root.json": {
            "version": 9,
            "len": 6766,
            "expiration": "12 Sep 24 06:53 UTC",
            "error": ""
        },
        "snapshot.json": {
            "version": 132,
            "len": 2302,
            "expiration": "09 Apr 24 16:16 UTC",
            "error": ""
        },
        "targets.json": {
            "version": 9,
            "len": 5478,
            "expiration": "12 Sep 24 06:13 UTC",
            "error": ""
        },
        "timestamp.json": {
            "version": 169,
            "len": 723,
            "expiration": "26 Mar 24 16:16 UTC",
            "error": ""
        }
    }
}
Error: Process completed with exit code 1.

[jenstroeger]

‼️ @ianlewis @laurentsimon please advise! This is breaking all our build pipelines, and talking with folks their pipelines are also dead in the water ‼️

[haydentherapper]

@laurentsimon @ianlewis You just need to bump Cosign to at least v2.2.0, latest is v2.2.3.

I'd also recommend checking your renovate settings, I would have expected this bump to get picked up automatically.

[behnazh-w]

@haydentherapper Cosign in SLSA Verifier needs to be updated to the latest version as well.

[suzuki-shunsuke]

haydentherapper Cosign in SLSA Verifier needs to be updated to the latest version as well.

According to the announcement, v2.2.0 should support a new TUF Trust root, so I don't think we need to update Cosign in SLSA Verifier.

[suzuki-shunsuke]

I see. The latest slsa-github-generato v1.9.0 uses the old slsa-verifier which uses the old cosign v2.0.2

https://github.com/slsa-framework/slsa-github-generator/blob/07e64b653f10a80b6510f4568f685f8b7b9ea830/.github/actions/generate-builder/action.yml#L95

https://github.com/slsa-framework/slsa-verifier/blob/c9abffe4d2ab2ffa0b2ea9b2582b84164f390adc/go.mod#L21

The main branch uses the latest slsa-verifier v2.4.1.

https://github.com/slsa-framework/slsa-github-generator/blob/b595e069f78187eaead9bac15429f87237be550a/.github/actions/generate-builder/action.yml#L95

So we need to release a new version of slsa-github-generator, then the issue would be solved.

[suzuki-shunsuke]

@laurentsimon @ianlewis I'm sorry to bother you, but could you release a new version of slsa-github-generator to solve this issue? https://github.com/slsa-framework/slsa-github-generator/issues/3350#issuecomment-2008731217

[laurentsimon]

Hi, yes let's do that. I think we need to revert a few breaking PRs we made recently in the slsa-github-generator. slsa-verifier should be ready to be released. @ramonpetgrave64 could you send PRs to temporarily revert the few breaking PRs we made recently?

[laurentsimon]

Note: slsa-verifier's latest version uses cosign 2.2.0 https://github.com/slsa-framework/slsa-verifier/blob/v2.4.1/go.mod so need not be updated.

[sgammon]

I see this is closed, but v1.9.0 is marked as the latest release... in the release notes update, it says:

error updating to TUF remote mirror: invalid This will occur only when generating provenance with all builders and generators. Affected versions: all versions up and including v1.9.0

Can this bug be reopened until a release occurs? Or is there a way to work around this, or can we use the generator at main?

cc / @laurentsimon @ianlewis

[laurentsimon]

Generator at main won't pass verification. We need to cut the release. @kpk47 is working on it

[sgammon]

Thank you @laurentsimon for being so responsive on this issue; I understand the commit auto-closed the bug. It's helpful to have something to follow. Good luck and thank you for your hard work securing software supply chains.

[jkreileder]

Works again for me with the un-finalized v1.10.0 pre-release.

[laurentsimon]

Release is available v1.10.0 https://github.com/slsa-framework/slsa-github-generator/releases/tag/v1.10.0

[ddworken]

I upgraded to v1.10.0 and that fixed generation of new provenance files, but I'm wondering: Is there a way to make old versions of slsa-verifier still work? From what I can tell, this change appears to make all old provenances unverifiable which seems like a significant problem.

[behnazh-w]

@ddworken The latest version of slsa-verifier is still is able to verify old provenances. Why do you need to use an old version of slsa-verifier?

@haydentherapper @bobcallaway Probably unrelated to this issue but what is the Sigstore/Cosign solution for revoked root keys and in general how does Sigstore provide backward compatibility with old versions of certificates, and therefore verifying existing legitimate provenances that were signed with revoked keys?

[ddworken]

The issue is that the old version of slsa-verifier isn't able to verify releases that were formerly valid. My project uses SLSA for releases and secure updates. I have daily tests running on Github Actions and it went from passing to failing in line with this bug:

You can see the test failure here. I also confirmed that this appears to apply to the main slsa-verifier binary, so this can easily be reproduced:

# From inside of the slsa-verifier git repo
# Check out the old version of slsa-verifier
git checkout v1.3.2
# Download a file and an old attestation
wget https://github.com/ddworken/hishtory/releases/download/v0.277/hishtory-linux-amd64
wget https://github.com/ddworken/hishtory/releases/download/v0.277/hishtory-linux-amd64.intoto.jsonl
# Run slsa-verifier
go run ./cli/slsa-verifier -artifact-path hishtory-linux-amd64 -provenance hishtory-linux-amd64.intoto.jsonl -source github.com/ddworken/hishtory

This fails with the output:

slsa-verifier output

``` Getting rekor entry error error verifying tlog entry: updating local metadata and targets: error updating to TUF remote mirror: invalid key remote status:{ "mirror": "https://sigstore-tuf-root.storage.googleapis.com", "metadata": { "root.json": { "version": 9, "len": 6766, "expiration": "12 Sep 24 06:53 UTC", "error": "" }, "snapshot.json": { "version": 132, "len": 2302, "expiration": "09 Apr 24 16:16 UTC", "error": "" }, "targets.json": { "version": 9, "len": 5478, "expiration": "12 Sep 24 06:13 UTC", "error": "" }, "timestamp.json": { "version": 170, "len": 721, "expiration": "29 Mar 24 16:08 UTC", "error": "" } } }: unable to fetch Rekor public keys from TUF repository, trying Redis search index to find entries by subject digest FAILED: SLSA verification failed: could not find a matching valid signature entry: got unexpected errors updating local metadata and targets: error updating to TUF remote mirror: invalid key remote status:{ "mirror": "https://sigstore-tuf-root.storage.googleapis.com", "metadata": { "root.json": { "version": 9, "len": 6766, "expiration": "12 Sep 24 06:53 UTC", "error": "" }, "snapshot.json": { "version": 132, "len": 2302, "expiration": "09 Apr 24 16:16 UTC", "error": "" }, "targets.json": { "version": 9, "len": 5478, "expiration": "12 Sep 24 06:13 UTC", "error": "" }, "timestamp.json": { "version": 170, "len": 721, "expiration": "29 Mar 24 16:08 UTC", "error": "" } } }: unable to fetch Rekor public keys from TUF repository: verifying tlog entry 24296fb24b8ad77a614df024f5e997739525746b275409b28df6a6b7285c7128f4f8e429a5cb1e42, updating local metadata and targets: error updating to TUF remote mirror: invalid key remote status:{ "mirror": "https://sigstore-tuf-root.storage.googleapis.com", "metadata": { "root.json": { "version": 9, "len": 6766, "expiration": "12 Sep 24 06:53 UTC", "error": "" }, "snapshot.json": { "version": 132, "len": 2302, "expiration": "09 Apr 24 16:16 UTC", "error": "" }, "targets.json": { "version": 9, "len": 5478, "expiration": "12 Sep 24 06:13 UTC", "error": "" }, "timestamp.json": { "version": 170, "len": 721, "expiration": "29 Mar 24 16:08 UTC", "error": "" } } }: unable to fetch Rekor public keys from TUF repository: verifying tlog entry 24296fb24b8ad77af7d9a9a7dbb2e7c25c5d46e41770969259f3d72fd20f0016bd5b1a54229bcbba exit status 2 ```

So it seems to me that slsa-verifier is now failing to verify attestations that used to pass. It is true that the latest version of slsa-verifier is able to verify old attestations, but this doesn't seem to help anyone who is stuck with an old version of slsa-verifier.

Why do you need to use an old version of slsa-verifier?

This is because my project embeds slsa-verifier in order to provide secure updates. So when slsa-verifier suddenly breaks in this way, it means that any clients with the old version of slsa-verifier become effectively stranded since they cannot upgrade to new versions.

[haydentherapper]

Probably unrelated to this issue but what is the Sigstore/Cosign solution for revoked root keys and in general how does Sigstore provide backward compatibility with old versions of certificates, and therefore verifying existing legitimate provenances that were signed with revoked keys?

@behnazh-w Supporting validity windows was the motivation behind the design of the trust root spec. For each root of trust, a validity time period is specified. If a compromise were to occur, rather than revoking the root material entirely and assuming we can determine the window when compromise occurred, the root material would simply be specified as valid only up to the compromise.

All Sigstore clients (eg sigstore-python, js, etc) but Cosign support ingesting the roots of trust in this format currently. Cosign will be updated to read it soon.

[laurentsimon]

So it seems to me that slsa-verifier is now failing to verify attestations that used to pass. It is true that the latest version of slsa-verifier is able to verify old attestations, but this doesn't seem to help anyone who is stuck with an old version of slsa-verifier.

You're correct. The TUF root updates need an update of slsa-verifier to v2.4.1. We could backport the fixes, but it would still require you to update your slsa-verifier version.

[ddworken]

So it seems to me that slsa-verifier is now failing to verify attestations that used to pass. It is true that the latest version of slsa-verifier is able to verify old attestations, but this doesn't seem to help anyone who is stuck with an old version of slsa-verifier.

You're correct. The TUF root updates need an update of slsa-verifier to v2.4.1. We could backport the fixes, but it would still require you to update your slsa-verifier version.

Got it, thanks for confirming! I'm wondering: Is this class of breakage something you expect to continue to need to happen with slsa-verifier? If so, it seems like this may be an important caveat to document (maybe under the "Known Isssues" heading) since having a previously working verifier continue to work seems like a reasonable expectation.

[laurentsimon]

We don't expect this to happen. @haydentherapper can comment on the specifics of cosign.

[haydentherapper]

Breaking changes are not something that is going to happen frequently and if they do, there will be a significant period of time to have clients migrate over.