Signed Kubewarden policies are show as not signed.

[jvanz]

Describe the bug Recently, the Kubewarden team released some policies signed with cosign. But artifacthub is not able to detect the signarture. Thus, the policies are show as not signed. However, we can verify the signatures using the cosign command:

cosign verify ghcr.io/kubewarden/policies/user-group-psp:v0.4.6 \
  --certificate-identity-regexp 'https://github.com/kubewarden/*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Expected behavior Mark the latest release of the Kubewarden policies as signed.

Screenshot

Additional context You can see the command used to sign the policies here.

[tegioz]

Hi @jvanz 👋

I've just run a test locally for that repo and the policy's signature was detected.

I've forced the same package in artifacthub.io to be processed now and the signature was detected as well.

Is it possible that the containers were signed after AH indexed that package version?

Please note that AH won't process an existing package version again unless its digest changes. So ideally, by the time AH detects a new version in the git repo, the containers should already be signed.

[viccuad]

Hi @tegioz, (Kubewarden maintainer here too)

Is it possible that the containers were signed after AH indexed that package version?

Yes, it's possible there was some minutes of difference, the time it takes to build, sign, and push the package in CI. Our approach is the one that has both the artifacthub-pkg.yml and artifacthub-repo.yml committed in the repo. Our process is:

1. Repo HEAD with new upcoming version (eg: 0.5.0), artifacthub-pkg.yml's version field being also 0.5.0.
2. Tag HEAD with v0.5.0. I suppose AH's webhook triggers here, and AH indexes the artifacthub-pkg.yml with the new version 0.5.0, or potentially AH indexed the version when the changed artifacthub-pkg.yml landed in the repo, much before the tag. There wasn't time yet for GHA to build, sign, and push the release, and we receive an e-mail from AH with "Something went wrong tracking repository user-group-psp".
3. GHA triggers with the tag, builds, signs, and push the release. This is done for example with this reusable workflow.

Please note that AH won't process an existing package version again unless its digest changes.

Honest question: did AH change behaviour here? I had the feeling that in the past it re-indexed packages with a cron. This problem is also happening for signed packages that have been released weeks prior, and I was hoping that AH would eventually re-index them and mark them as signed.

Example of another Kubewarden policy, released a while ago, that is marked also as unsigned: https://artifacthub.io/packages/kubewarden/verify-image-signatures/verify-image-signatures

cosign verify ghcr.io/kubewarden/policies/verify-image-signatures:v0.2.5 \
  --certificate-identity-regexp 'https://github.com/kubewarden/*' \
   --certificate-oidc-issuer https://token.actions.githubusercontent.com

[tegioz]

Hi @viccuad 👋

Question: does AH re-index packages after a while?

No, it doesn't, unless we receive a signal they've changed. There are >178k releases at the moment and growing, we'd run into rate limiting problems soon 😅 And honestly, even if it did, I don't think it'd look nice to display a package unsigned for one day or week, when it's actually signed from the beginning. We have the digest at both the repository and package level to signal the need for processing, and we ping repos every 30 mins to detect those changes. AH trackers run twice an hour, at mins 0 and 30 approximately, so depending on when your workflows run, this will likely be a problem often.

Some possible workarounds off the top of my head (none super nice TBH, trying to think harder 💆🏻‍♂️):

• Use a different branch to publish on AH. The release workflow could cherry pick commits from the source branch at the end (when containers have already been signed).
• Update digest in AH metadata once the release workflow is done (this one is worse IMHO, as even it'd work, it'd require AH to always process packages twice, and it'd appear unsigned for at least 30 mins).

[tegioz]

Honest question: did AH change behaviour here? I had the feeling that in the past it re-indexed packages with a cron.

I think you edited while I was replying, hope to not have missed anything 🙂 No, this hasn't changed, it's always worked like that (still a cron job though 😉). It's just that it depends on the timing like I mentioned in my previous comment.

[tegioz]

3. GHA triggers with the tag, builds, signs, and push the release.

Please note that this can be a problem beyond the signature detection: we may be publishing a package on AH that might actually not exist yet [1] (for a few seconds, minutes, or longer if GHA is slow or down). It should be most of the time just a few seconds or minutes, but users could try to install the package following your instructions as soon as they receive the new release notification from AH.

[1] (edit): the container image may not have been built yet.

[viccuad]

Thanks @tegioz!

Been thinking about workarounds, and the best one for our fits is using a different branch to publish to AH. WIll solve on our end that way.

This can be closed :).

[tegioz]

Awesome, no worries!