search

kubernetes / release

Release infrastructure for Kubernetes and related components
Apache License 2.0
485 stars 503 forks source link

Signing flakes on krel release #2809

Open jeremyrickard opened 1 year ago

jeremyrickard commented 1 year ago

What happened:

On a couple of the patch releases, we hit flakes with signing and needed to re-run the no mock release stages:

Signing Flake for 1.24.9

level=fatal msg="signing the blobs: signing the file /tmp.k8s/release-sign-blobs-1616671436/kubernetes-release/stage/v1.24.9-rc.0.42+15da17757dcdd3/v1.24.9/gcs-stage/v1.24.9/bin/windows/amd64/kubectl.exe: verifying signed file: /tmp.k8s/release-sign-blobs-1616671436/kubernetes-release/stage/v1.24.9-rc.0.42+15da17757dcdd3/v1.24.9/gcs-stage/v1.24.9/bin/windows/amd64/kubectl.exe: open /tmp.k8s/release-sign-blobs-1616671436/kubernetes-release/stage/v1.24.9-rc.0.42+15da17757dcdd3/v1.24.9/gcs-stage/v1.24.9/bin/windows/amd64/kubectl.exe.cert: no such file or directory"

Signing Flake on 1.23.15

level=fatal msg="signing the blobs: signing the file /tmp.k8s/release-sign-blobs-691105010/kubernetes-release-gcb/stage/v1.23.15-rc.0.32+06089cc90f824e/v1.23.15/gcs-stage/v1.23.15/bin/windows/arm64/kubectl-convert.exe: verifying signed file: /tmp.k8s/release-sign-blobs-691105010/kubernetes-release-gcb/stage/v1.23.15-rc.0.32+06089cc90f824e/v1.23.15/gcs-stage/v1.23.15/bin/windows/arm64/kubectl-convert.exe: open /tmp.k8s/release-sign-blobs-691105010/kubernetes-release-gcb/stage/v1.23.15-rc.0.32+06089cc90f824e/v1.23.15/gcs-stage/v1.23.15/bin/windows/arm64/kubectl-convert.exe.cert: no such file or directory"

Signing Flake on 1.22.17

level=fatal msg="signing the blobs: signing the file /tmp.k8s/release-sign-blobs-3048792261/kubernetes-release/stage/v1.22.17-rc.0.16+611514908b25d5/v1.22.17/gcs-stage/v1.22.17/bin/windows/amd64/kubelet.exe: verifying signed file: /tmp.k8s/release-sign-blobs-3048792261/kubernetes-release/stage/v1.22.17-rc.0.16+611514908b25d5/v1.22.17/gcs-stage/v1.22.17/bin/windows/amd64/kubelet.exe: open /tmp.k8s/release-sign-blobs-3048792261/kubernetes-release/stage/v1.22.17-rc.0.16+611514908b25d5/v1.22.17/gcs-stage/v1.22.17/bin/windows/amd64/kubelet.exe.cert: no such file or directory"

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

saschagrunert commented 1 year ago

@cpanato this seems to be another race we hit when signing release artifacts. Do you want to give this a look? (maybe @puerco already did)

xmudrii commented 1 year ago

/remove-label priority/important-soon /priority critical-urgent

k8s-ci-robot commented 1 year ago

@xmudrii: The label(s) /remove-label priority/important-soon cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to [this](https://github.com/kubernetes/release/issues/2809#issuecomment-1402125833): >/remove-label priority/important-soon >/priority critical-urgent Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
saschagrunert commented 1 year ago

@kubernetes/release-managers Carlos is out for a couple of days, do you we have any volunteer to support here?

saschagrunert commented 1 year ago

First investigation: The certificate (.cert) file has to be written in cosign, after writing the signature: https://github.com/sigstore/cosign/blob/d1c6336475b4be26bb7fb52d97f56ea0a1767f9f/cmd/cosign/cli/sign/sign_blob.go#L120-L129

It looks like that we never come to the point where the file has to be written, so I'm assuming that len(rekorBytes) == 0 :thinking:

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

xmudrii commented 1 year ago

/remove-lifecycle stale

matglas commented 1 year ago

/assign

matglas commented 1 year ago

@jeremyrickard would it be possible to see more of the logs? I can't access them with the links above.

I am trying to create some context for myself to understand where in the process this happens. Is this the part triggered by krel release? If so where during that part are we doing a blob sign.

k8s-triage-robot commented 10 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

xmudrii commented 10 months ago

/remove-lifecycle stale

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

xmudrii commented 7 months ago

/lifecycle frozen