erwbgy
commented
2 years ago An example (using Contour 1.21.0) where the Status says that the Secret does not exist but it does:
$ kubectl describe httpproxy global | grep Message
Message: Spec.VirtualHost.TLS Secret "newtls" is invalid: Secret not found
Message: At least one error present, see Errors for details
$ kubectl get secret "newtls"
NAME TYPE DATA AGE
newtls kubernetes.io/tls 2 3h24m According to #3496 this can also be because the Secret is considered invalid but I don't see anything obviously wrong with it:
$ kubectl describe secret "newtls"
Name: newtls
Namespace: ...
Labels: <none>
Annotations: cert-manager.io/alt-names:
...
cert-manager.io/certificate-name: global
cert-manager.io/common-name: ...
cert-manager.io/ip-sans:
cert-manager.io/issuer-group: cert-manager.io
cert-manager.io/issuer-kind: ClusterIssuer
cert-manager.io/issuer-name: ...
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
tls.crt: 6245 bytes
tls.key: 1679 bytes
$ kubectl get secret newtls -o json | jq -r '.data["tls.crt"]' | base64 -d | showcerts2 /dev/stdin
---
X.509v3 TLS Certificate (RSA 2048) [Serial: 6690...9970]
...
Valid from: 2022-06-23T21:40:49Z
to: 2024-06-22T21:40:49Z
---
X.509v3 Intermediate CA Certificate (RSA 2048) [Serial: 1605...1715]
...
Valid from: 2015-10-07T15:22:23Z
to: 2025-10-07T15:32:23Z
---
X.509v3 Root CA Certificate (RSA 4096) [Serial: 2177...9169]
...
Valid from: 2015-10-07T14:06:52Z
to: 2035-10-07T14:11:56ZContour logs:
$ stern contour- | grep -i newtls
+ contour-786b48f464-hz9h4 › contour
+ contour-786b48f464-hfppz › contour
contour-786b48f464-hfppz contour time="2022-06-23T21:50:56Z" level=error msg="invalid TLS certificate: failed to locate certificate" context=KubernetesCache kind=Secret name=newtls namespace=... version=v1
contour-786b48f464-hfppz contour time="2022-06-24T09:17:25Z" level=error msg="stream terminated" connection=1114 context=xds error="context canceled" node_id=envoy-d6845759f-bxjrl node_version=v1.22.0 resource_names="[.../newtls/ced75e6a4e]" response_nonce=1634 type_url=type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret version_info=1634
tsaarni
youngnick
What steps did you take and what happened:
Create an HTTPProxy, Secret, Service and Deployment - status of HTTPProxy is valid as expected. A few days later delete the HTTPProxy, Secret, Service and Deployment, then only recreate the HTTPProxy, Service and Deployment - status of HTTPProxy goes to Valid which is NOT expected as the Secret object no longer exists. It appears that the previous Secret state is cached and not updated.
We have also seen numerous cases of the opposite problem where an HTTPProxy shows as invalid complaining about a missing Secret when it does actually exist.
This does not always happen but in one relaively busy cluster we see issues consistently after a few days. Restarting Contour is the only way to resolve these issues.
What did you expect to happen:
HTTPProxy Status should accurately reflect the existence of the Secret object.
Anything else you would like to add:
Contour logs show an XDS error for the Secret where the status is incorrect:
Not sure if it is related but running contour with --debug I see a continuous stream of messages from the leader:
Environment:
kubectl version): v1.21.8+ed4d8fd/etc/os-release): Red Hat Enterprise Linux CoreOS 48.84.202204202010-0