cert-manager / trust-manager

trust-manager is an operator for distributing trust bundles across a Kubernetes cluster.
https://cert-manager.io/docs/projects/trust-manager/
Apache License 2.0
243 stars 65 forks source link

Incorrect error handling in cert-manager-package-debian updater #246

Closed SgtCoDFish closed 8 months ago

SgtCoDFish commented 8 months ago

We have a CI process which regularly tries to keep our Debian package up to date by setting up a fresh debian container and checking the latest version of ca-certificates against the latest tag available for our package bundles.

The logic is in this file.

The offending line is:

https://github.com/cert-manager/trust-manager/blob/620be4f7184d2f5cea8fcae604a4c1e1edceff06/hack/update-debian-trust-package.sh#L93

This line treats a nonzero exit code as evidence that the upstream version doesn't exist. However, that's only true if crane got a 404. If quay.io is down when we try to check the latest version, we'll see (say) a 504 error but the exit code will still be interpreted as the package not existing.

This happened on 2023-11-27. Crane got the following 504 error:

2023/11/27 16:35:18 HEAD request failed, falling back on GET: GET https://quay.io/v2/: unexpected status code 504 Gateway Timeout: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.20.1</center>
</body>
</html>
Error: GET https://quay.io/v2/: unexpected status code 502 Bad Gateway: <!doctype html>
<html lang=en>
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to the target URL: <a href="https://quay.io/quay-registry/static/502.html/">https://quay.io/quay-registry/static/502.html/</a>. If not, click the link.

+++ latest image appears not to exist; building and pushing quay.io/jetstack/cert-manager-package-debian:20210119.0

And following this, the trust package was rebuilt.

The error handling should be improved so that we specifically look for a "not found" response.