Closed wSedlacek closed 4 months ago
ReToCode
commented
8 months ago I'm assuming you have enabled some of the experimental encryption features? In 1.13 certificates are created by cert-manager (via net-certmanager) instead of an internal reconciler. Can you try to delete the existing secret: knative-serving/routing-serving-certs so it will be re-populated by a Secret from cert-manager?
The one from net-istio should be fine as it is newly introduced.
wSedlacek
commented
8 months ago I'm assuming you have enabled some of the experimental encryption features?
As far as I know I haven't done this.
To be verbose on the complete setup I have a pulumi build script setup that
kube-prometheus-stack helm chartbase, istiod, gateway helm chartsknative-system namespace)issuerRef to the Google Trust Services ClusterIssuer I applied previously and the systemInternalIssuerRef to the knative-selfsigned-issuer as shown in the _exampleI am doing this on a GKE standard cluster that was brand new for testing this build script. I have torn it down and reapplied it several times. I get pretty consistent results. On knative 1.12.0 I don't get the loop of certificates trying to be made, but on knative 1.13.0 I do.
Here are my configurations for KnativeServing and KnativeEventing
apiVersion: operator.knative.dev/v1beta1
kind: KnativeEventing
metadata:
name: knative-eventing
namespace: knative-eventing
spec:
version: {{.Values.version}}
source:
redis:
enabled: true
---
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
name: knative-serving
namespace: knative-serving
spec:
version: {{.Values.version}}
config:
network:
external-domain-tls: {{.Values.https | ternary "Enabled" "Disabled"}}
domain:
'example.com': ''Can you try to delete the existing secret: knative-serving/routing-serving-certs so it will be re-populated by a Secret from cert-manager?
I do see this secret as already existing, however it is listed as controlled by the KnativeServing operator. I can manually delete it and it seems not not be immediately recreated by the operator. It does allow the certificate to be issued after that which of course recreates the secret by controlled by the Certificate. I suspect though when making a change to the KnativeServing operator like updating the version it would reapply this secret and bring the issue back.
So I think the root of the problem is that both the KnativeServing operator and the cert-manager Certificate are trying to control the same secret.
For my build script this is also problematic as both the Secret controlled by KnativeServing and the Certificate are not directly created from my build script so I would need to wait for that Secret to exists from KnativeServing before I can manually delete it for the Certificate to get issued.
Is there a way to make the KnativeServing operator not create routing-serving-certs when using net-certmaanger?
Is there a way to make the certificate created by the KnativeServing operator able to be overwritten by the one created by cert-manager without getting stuck in a loop?
ReToCode
commented
8 months ago I'm still not sure why you are facing this. Let me point to the important parts in code:
In release-1.13 you should have:
knative-serving called routing-serving-certs (It is possible that it is owned by the operator, when you use that to install Knative Serving) --> https://github.com/knative/serving/blob/release-1.13/config/core/300-secret.yaml#L42. This Secret should not be touched by net-certmanagerCertificate and a KnativeCertificate object in istio-system from https://github.com/knative-extensions/net-istio/blob/release-1.13/config/700-istio-knative-certificate.yaml#L23. It has the same name, but is not the same as the one in knative-serving. This certificate is read by net-certmanager and populated by cert-managerSo something is off here, as your error says:
Reconciling Cert-Manager certificate for Knative cert.
knative.dev/kind networking.internal.knative.dev.Certificate
knative.dev/key: knative-serving/routing-serving-certsSo I think the root of the problem is that both the KnativeServing operator and the cert-manager Certificate are trying to control the same secret.
In 1.14 we will have a KnativeCertificate to replace the Secret, but not in 1.13. So there should not be a conflict. Can you post the following outputs?
kubectl get kcert -A
kubectl get certificate -A
kubectl get secret -n knative-serving
kubectl get secret -n istio-system (or your istio system namespace if you override it)Also, do you use the default istio-system namespace or did you change it?
wSedlacek
commented
8 months ago Also, do you use the default istio-system namespace or did you change it?
Yes I have istio-system as my namespace for istio AND the istio-ingressgateway
Can you post the following outputs?
kubectl get kcert -A
NAMESPACE NAME READY REASON
knative-serving routing-serving-certs False SecretMismatchNote: routing-serving-certs cycles between Ready True and False
kubectl get certificate -A
NAMESPACE NAME READY SECRET AGE
cert-manager knative-selfsigned-ca True knative-selfsigned-ca 98s
knative-serving routing-serving-certs False routing-serving-certs 92sNote: routing-serving-certs cycles between Ready True and False
kubectl get secret -n knative-serving
NAME TYPE DATA AGE
knative-serving-certs Opaque 6 2m46s
net-certmanager-webhook-certs Opaque 3 2m27s
net-istio-webhook-certs Opaque 3 2m39s
routing-serving-certs Opaque 6 2m46s
routing-serving-certs-dwgxb Opaque 1 0s
serving-certs-ctrl-ca Opaque 4 2m47s
webhook-certs Opaque 3 2m41kubectl get secret -n istio-system
NAME TYPE DATA AGE
istio-ca-secret istio.io/ca-root 5 22mAs above , you should not have a kcert in namespace knative-serving only a Secret (see our release manifests). But you should have that one istio-system which you do not. So something is off with your installation YAML. Can you check the manifest that your installation produces and see where that comes from?
It looks like this file is getting placed in knative-serving somehow.
wSedlacek
commented
7 months ago I have been a bit busy with other projects but I am finally getting back to this. Could it be that this function is being applied incorrectly when the operator is used to install knative? https://github.com/knative/operator/blob/d6ee177ea432bead00717544aedde5ead10680a8/pkg/reconciler/knativeserving/knativeserving.go#L150-L154
It looks like the extra argument gives extra behavior for the ingress, does that need a new entry added to handle the kcert so that it gets put in the right namespace?
wSedlacek
commented
4 months ago This bug does not occur in 1.14.0 as expected, and the issue with net-certmanager is completely removed with 1.14.1 because it is now integrated into serving. All good now! Thank you!
Describe the bug When applying
https://github.com/knative/net-certmanager/releases/download/knative-v1.13.0/release.yamlit tries to create a certificate (self-signed) forkn-routingbut gets stuck in a loop.Specifically
SecretMismatch Secret contains a private key that does not match the current CertificateRequestExpected behavior
I am not entirely sure why it is getting stuck in a loop or if the place to change it is in
net-certmanagerhoweverkn-routingseems to have been introduced in1.13innet-istioso this seemed like the place to start.Knative release version 1.13.0