The status of Route/Ksvc is not updated even when KCert is updated

nak3 commented 2 years ago

In what area(s)?

/area networking

What version of Knative?

HEAD

Steps to Reproduce the Problem

1. Deploy Knative with autoTLS.

istioctl install -y
kubectl apply -f ${SERVING_REPO}/third_party/cert-manager-latest/

kubectl apply --filename https://storage.googleapis.com/knative-nightly/serving/latest/serving-crds.yaml
kubectl apply --filename https://storage.googleapis.com/knative-nightly/serving/latest/serving-core.yaml
kubectl apply --filename https://storage.googleapis.com/knative-nightly/net-istio/latest/release.yaml

kubectl create ns serving-tests
kubectl apply -f ${SERVING_REPO}/test/config/autotls/certmanager/caissuer/
kubectl patch cm config-network -n "knative-serving" -p '{"data":{"autoTLS":"Enabled"}}'

2. Modify `config-certmanager` with the invalid value

Note: This is a simple step to produce the issue. The same issue happens with some other reasons like let's encrypt issued the certificates slowly.

$ kubectl edit cm -n knative-serving  config-certmanager

BEFORE:

  issuerRef: |
    kind: ClusterIssuer
    name: ca-issuer

AFTER

  issuerRef: |
    kind: ClusterIssuer
    name: INVALID

3. Create Ksvc (it hangs due to invalid cert)

$ kn service create hello-example --image=gcr.io/knative-samples/helloworld-go
  ...
  3.607s Certificate default.example.com is not ready.

The status of Ksvc and route are CertificateNotReady as expected.

$ kubectl get ksvc,rt
NAME                                        URL                                         LATESTCREATED         LATESTREADY           READY     REASON
service.serving.knative.dev/hello-example   https://hello-example.default.example.com   hello-example-00001   hello-example-00001   Unknown   CertificateNotReady

NAME                                      URL                                         READY     REASON
route.serving.knative.dev/hello-example   https://hello-example.default.example.com   Unknown   CertificateNotReady

4. Fix the cert by reverting step-2

$ kubectl edit cm -n knative-serving  config-certmanager

AFTER

  issuerRef: |
    kind: ClusterIssuer
    name: ca-issuer

5. Check the status

Kcert and King become Ready.

$ kubectl get kcert,king
NAME                                                              READY   REASON
certificate.networking.internal.knative.dev/default.example.com   True    

NAME                                                    READY   REASON
ingress.networking.internal.knative.dev/hello-example   True

But Ksvc and Route are still CertificateNotReady.

$ kubectl get ksvc
NAME            URL                                         LATESTCREATED         LATESTREADY           READY     REASON
hello-example   https://hello-example.default.example.com   hello-example-00001   hello-example-00001   Unknown   CertificateNotReady

Expected Behavior

When Kcert and Kingress became ready at step-4, Ksvc should become Ready as well.

Actual Behavior

The status of Ksvc is not updated at step-4.
When some events trigger the reconcile loop in controller, the status will be updated.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

zhilingc commented 2 years ago

+1 same issue :/ On knative-serving 1.3.2.

richardvflux commented 1 year ago

I'm on knative-serving 1.9.2 and I'm getting this as well.


Name:         test.dlvr-test.dlvr-test.fluxfederation.com
Namespace:    test
Labels:       networking.knative.dev/wildcardDomain=dlvr-test.dlvr-test.fluxfederation.com
Annotations:  networking.knative.dev/certificate.class: cert-manager.certificate.networking.knative.dev
API Version:  networking.internal.knative.dev/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2023-04-05T20:42:18Z
  Generation:          1
  Managed Fields:
    API Version:  networking.internal.knative.dev/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:networking.knative.dev/certificate.class:
        f:labels:
          .:
          f:networking.knative.dev/wildcardDomain:
        f:ownerReferences:
          .:
          k:{"uid":"72ed00fb-17a0-47b9-8dc3-d8b3640f3685"}:
      f:spec:
        .:
        f:dnsNames:
        f:secretName:
    Manager:      controller
    Operation:    Update
    Time:         2023-04-05T20:42:18Z
    API Version:  networking.internal.knative.dev/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:notAfter:
        f:observedGeneration:
    Manager:      controller
    Operation:    Update
    Subresource:  status
    Time:         2023-04-05T20:42:28Z
  Owner References:
    API Version:           v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Namespace
    Name:                  test
    UID:                   72ed00fb-17a0-47b9-8dc3-d8b3640f3685
  Resource Version:        29874350
  UID:                     4b7db946-1f9a-4296-839d-349ca9752309
Spec:
  Dns Names:
    *.test.dlvr-test.dlvr-test.fluxfederation.com
  Secret Name:  test.dlvr-test.dlvr-test.fluxfederation.com
Status:
  Conditions:
    Last Transition Time:  2023-04-05T20:42:28Z
    Status:                True
    Type:                  Ready
  Not After:               2023-07-04T19:42:22Z
  Observed Generation:     1
Events:
  Type    Reason   Age   From                    Message
  ----    ------   ----  ----                    -------
  Normal  Created  16m   namespace-controller    Created Knative Certificate test/test.dlvr-test.dlvr-test.fluxfederation.com
  Normal  Created  16m   certificate-controller  Created Cert-Manager Certificate test/test.dlvr-test.dlvr-test.fluxfederation.com```

```get route -n test
NAME       URL                                                            READY     REASON
my-nginx   https://my-nginx.test.dlvr-test.dlvr-test.fluxfederation.com   Unknown   CertificateNotReady`

This happened to our entire cluster when we upgraded from 1.7.2 -> 1.9.2, we ended up having to manually delete all the kservices and getting Argo to recreate them which causes the resolve to work. ```

mpluhar commented 9 months ago

Any update on this issue?

knative / serving