namespace-wildcard-cert-selector with empty object does not work

streamnsight commented 2 years ago

/area networking /kind spec

What version of Knative?

knative-serving v1.2.5

Expected Behavior

the doc for config-networking states:

# namespace-wildcard-cert-selector specifies a LabelSelector which
    # determines which namespaces should have a wildcard certificate
    # provisioned.
    #
    # Use an empty value to disable the feature (this is the default):
    #   namespace-wildcard-cert-selector: ""
    #
    # Use an empty object to enable for all namespaces
    #   namespace-wildcard-cert-selector: {}
    #
    # Useful labels include the "kubernetes.io/metadata.name" label to
    # avoid provisioning a certifcate for the "kube-system" namespaces.
    # Use the following selector to match pre-1.0 behavior of using
    # "networking.knative.dev/disableWildcardCert" to exclude namespaces:
    #
    # matchExpressions:
    # - key: "networking.knative.dev/disableWildcardCert"
    #   operator: "NotIn"
    #   values: ["true"]
    namespace-wildcard-cert-selector: ""

especally:

Use an empty object to enable for all namespaces
    #   namespace-wildcard-cert-selector: {}

Actual Behavior

when using an empty object as suggested, I get a validation error:

# configmaps "config-network" was not valid:
# * <nil>: Invalid value: "The edited file failed validation": ValidationError(ConfigMap.data.namespace-wildcard-cert-selector): invalid type for io.k8s.api.core.v1.ConfigMap.data: got "map", expected "string"

I'm trying to use a specific label but it doesn't seem to work, however it is not clear what object is being looked up therefore it is not clear what label should be used.

Steps to Reproduce the Problem

Edit the config-network and set namespace-wildcard-cert-selector to {}

Ideally I want to allow this for all namespaces, but this setting doesn't work.

streamnsight commented 2 years ago

looking through the code I realize the correct syntax is actually to use:

namespace-wildcard-cert-selector: '{}'

but even when I do that, the Challenge being submitted is not for a wildcard certificate.

skonto commented 2 years ago

cc @nak3

streamnsight commented 2 years ago

@nak3 any insight on why it doesn't work?

nak3 commented 2 years ago

Sorry I am not available until the end of Oct.

cc @evankanderson @dprotaso Do you have any idea?

evankanderson commented 2 years ago

I'm not sure, I can try a test on my own cluster later today.

evankanderson commented 2 years ago

Are you using a ClusterIssuer that supports DNS01 challenges?

streamnsight commented 2 years ago

@evankanderson yes, I am. I tested with direct requests, and when i ask for *.domain I get a wildcard cert. but as pointed above, the challenge request generated is not for a *.domain, it is for the fully qualified subdomain, so it is in the request being made that there is an issue. Does it work for you?

streamnsight commented 2 years ago

@nak3 @evankanderson any news on this? I'd like to hear someone confirm they are able to make it work and try to reproduce, because it doesn't work for me.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

dprotaso commented 1 year ago

/reopen /triage accepted

knative-prow[bot] commented 1 year ago

@dprotaso: Reopened this issue.

In response to [this](https://github.com/knative/serving/issues/13359#issuecomment-1481056832): >/reopen >/triage accepted Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

streamnsight commented 1 year ago

@dprotaso @nak3 @evankanderson Hi guys, I'm happy to test some stuff, try some config changes, or even pull the repo and try some code changes, but I'd like to know where to start. The CSR does not request a wildcard cert.

Here is my config (annotations etc... trimmed for clarity):

apiVersion: v1
data:
  issuerRef: |
    kind: ClusterIssuer
    name: letsencrypt-prod-wildcard-issuer
kind: ConfigMap
metadata:
  name: config-certmanager
  namespace: knative-serving

Note my wildcard issuer works fine for the main cert

apiVersion: v1
data:
  auto-tls: Enabled
  domain-template: '{{ index .Labels "serving.kserve.io/inferenceservice" }}.{{.Namespace}}.{{.Domain}}'
  http-protocol: Redirected
  namespace-wildcard-cert-selector: '{}'
kind: ConfigMap
metadata:
  name: config-network
  namespace: knative-serving

The CertificateRequest generated looks like this:

Name:         route-fc295302-60b8-4d24-aa1a-08d42702249c-6c77k
Namespace:    epl
Labels:       serving.knative.dev/route=mnist-e2eab-predictor-default
Annotations:  cert-manager.io/certificate-name: route-fc295302-60b8-4d24-aa1a-08d42702249c
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: route-fc295302-60b8-4d24-aa1a-08d42702249c-6fhfm
              networking.knative.dev/certificate.class: cert-manager.certificate.networking.knative.dev
              serving.knative.dev/creator: system:serviceaccount:kubeflow:kserve-controller-manager
              serving.knative.dev/lastModifier: system:serviceaccount:kubeflow:kserve-controller-manager
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2023-04-11T14:14:51Z
  Generate Name:       route-fc295302-60b8-4d24-aa1a-08d42702249c-
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
          .:
          k:{"type":"Approved"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
    Manager:      cert-manager-certificaterequests-approver
    Operation:    Update
    Subresource:  status
    Time:         2023-04-11T14:14:51Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-name:
          f:cert-manager.io/certificate-revision:
          f:cert-manager.io/private-key-secret-name:
          f:networking.knative.dev/certificate.class:
          f:serving.knative.dev/creator:
          f:serving.knative.dev/lastModifier:
        f:generateName:
        f:labels:
          .:
          f:serving.knative.dev/route:
        f:ownerReferences:
          .:
          k:{"uid":"2e072522-8452-4247-9ffa-8222b8355cd7"}:
      f:spec:
        .:
        f:issuerRef:
          .:
          f:kind:
          f:name:
        f:request:
    Manager:      cert-manager-certificates-request-manager
    Operation:    Update
    Time:         2023-04-11T14:14:51Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:certificate:
        f:conditions:
          k:{"type":"Ready"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
    Manager:      cert-manager-certificaterequests-issuer-acme
    Operation:    Update
    Subresource:  status
    Time:         2023-04-11T14:20:59Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  route-fc295302-60b8-4d24-aa1a-08d42702249c
    UID:                   2e072522-8452-4247-9ffa-8222b8355cd7
  Resource Version:        6275349
  UID:                     293d31a8-0135-4196-ba8f-419fc54498bd
Spec:
  Extra:
    authentication.kubernetes.io/pod-name:
      cert-manager-588d96b87f-trvr2
    authentication.kubernetes.io/pod-uid:
      328d65a0-1a5f-4699-b628-812e36bd5a28
  Groups:
    system:serviceaccounts
    system:serviceaccounts:cert-manager
    system:authenticated
  Issuer Ref:
    Kind:    ClusterIssuer
    Name:    letsencrypt-prod-wildcard-issuer
  Request:   [...]
  UID:       aed1d589-adbb-4e2a-85d2-4f1c1a50cd22
  Username:  system:serviceaccount:cert-manager:cert-manager
Status:
  Certificate:  ...
  Conditions:
    Last Transition Time:  2023-04-11T14:14:51Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2023-04-11T14:20:59Z
    Message:               Certificate fetched from issuer successfully
    Reason:                Issued
    Status:                True
    Type:                  Ready
Events:
  Type    Reason             Age   From                                          Message
  ----    ------             ----  ----                                          -------
  Normal  cert-manager.io    36m   cert-manager-certificaterequests-approver     Certificate request has been approved by cert-manager.io
  Normal  OrderCreated       36m   cert-manager-certificaterequests-issuer-acme  Created Order resource epl/route-fc295302-60b8-4d24-aa1a-08d42702249c-6c77k-160966232
  Normal  OrderPending       36m   cert-manager-certificaterequests-issuer-acme  Waiting on certificate issuance from order epl/route-fc295302-60b8-4d24-aa1a-08d42702249c-6c77k-160966232: ""
  Normal  CertificateIssued  30m   cert-manager-certificaterequests-issuer-acme  Certificate fetched from issuer successfully

The challenge looks like this:

Name:         route-fc295302-60b8-4d24-aa1a-08d42702249c-6c77k-160966232
Namespace:    epl
Labels:       serving.knative.dev/route=mnist-e2eab-predictor-default
Annotations:  cert-manager.io/certificate-name: route-fc295302-60b8-4d24-aa1a-08d42702249c
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: route-fc295302-60b8-4d24-aa1a-08d42702249c-6fhfm
              networking.knative.dev/certificate.class: cert-manager.certificate.networking.knative.dev
              serving.knative.dev/creator: system:serviceaccount:kubeflow:kserve-controller-manager
              serving.knative.dev/lastModifier: system:serviceaccount:kubeflow:kserve-controller-manager
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2023-04-11T14:14:51Z
  Generation:          1
  Managed Fields:
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-name:
          f:cert-manager.io/certificate-revision:
          f:cert-manager.io/private-key-secret-name:
          f:networking.knative.dev/certificate.class:
          f:serving.knative.dev/creator:
          f:serving.knative.dev/lastModifier:
        f:labels:
          .:
          f:serving.knative.dev/route:
        f:ownerReferences:
          .:
          k:{"uid":"293d31a8-0135-4196-ba8f-419fc54498bd"}:
      f:spec:
        .:
        f:commonName:
        f:dnsNames:
        f:issuerRef:
          .:
          f:kind:
          f:name:
        f:request:
    Manager:      cert-manager-certificaterequests-issuer-acme
    Operation:    Update
    Time:         2023-04-11T14:14:51Z
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:authorizations:
        f:certificate:
        f:finalizeURL:
        f:state:
        f:url:
    Manager:      cert-manager-orders
    Operation:    Update
    Subresource:  status
    Time:         2023-04-11T14:20:59Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  route-fc295302-60b8-4d24-aa1a-08d42702249c-6c77k
    UID:                   293d31a8-0135-4196-ba8f-419fc54498bd
  Resource Version:        6275345
  UID:                     50e3c05e-95c1-44d9-80e0-34cd4d672ffc
Spec:
  Common Name:  mnist-e2eab.epl.ocikubeflow.ml
  Dns Names:
    mnist-e2eab.epl.ocikubeflow.ml
  Issuer Ref:
    Kind:   ClusterIssuer
    Name:   letsencrypt-prod-wildcard-issuer
  Request:  [...]
Status:
  Authorizations:
    Challenges:
      Token:        rsIvFt2uHmmATgvIC3ChCY_TxwuvcGgZAWXbWyrEVRs
      Type:         http-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/218560382057/NPt-MA
      Token:        rsIvFt2uHmmATgvIC3ChCY_TxwuvcGgZAWXbWyrEVRs
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/218560382057/nAXMpw
      Token:        rsIvFt2uHmmATgvIC3ChCY_TxwuvcGgZAWXbWyrEVRs
      Type:         tls-alpn-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/218560382057/dGakIg
    Identifier:     mnist-e2eab.epl.ocikubeflow.ml
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/218560382057
    Wildcard:       false
  Certificate:      [...]
  Finalize URL:     https://acme-v02.api.letsencrypt.org/acme/finalize/1052399957/175543041917
  State:            valid
  URL:              https://acme-v02.api.letsencrypt.org/acme/order/1052399957/175543041917
Events:
  Type    Reason    Age   From                 Message
  ----    ------    ----  ----                 -------
  Normal  Created   34m   cert-manager-orders  Created Challenge resource "route-fc295302-60b8-4d24-aa1a-08d42702249c-6c77k-160-2518886456" for domain "mnist-e2eab.epl.ocikubeflow.ml"
  Normal  Complete  28m   cert-manager-orders  Order completed successfully

The CSR decoded looks like:

Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: CN=mnist-e2eab.epl.ocikubeflow.ml
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    [...]
                Exponent: 65537 (0x10001)
        Attributes:
        Requested Extensions:
            X509v3 Subject Alternative Name: 
                DNS:mnist-e2eab.epl.ocikubeflow.ml
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
    Signature Algorithm: sha256WithRSAEncryption
         [...]

and in all this you can see that the CN, the DNS alt name is NOT using the '*' wildcard, so that the cert issues in the end is a regular 2nd level subdomain cert, not a wildcard cert to the 1st level subdomain

Maybe I am misunderstanding this, but my expectation is that I get a cert for the 1st level subdomain the first time I deploy in this namespace, so that i don't need to get a new cert for each single endpoint when deploying (which takes some time and delays deployment)

Can someone confirm this is the intended way this works, and if it is, where should we look to figure why it doesn't.

Thanks

joeferraro commented 1 year ago

I've run into the same and I'm pretty sure it's due to specifying a custom domain-template. Wish this could get patched

nak3 commented 1 year ago

You need to domain-template for the case where label is missed with the default template{{ .Name }}.{{ .Namespace }}.{{ .Domain }}. For example in your case,

  domain-template: '{{if index .Labels "serving.kserve.io/inferenceservice" }}{{index .Labels "serving.kserve.io/inferenceservice" }}{{ else }}{{ .Name }}{{ end }}.{{ .Namespace }}.{{ .Domain }}'

Then, the wildcard cert creates the namespace with <namespace>.<domain> like:

$ kubectl get kcert --all-namespaces 
NAMESPACE            NAME                                         READY   REASON
bar                  bar.example.com                              True    
cert-manager         cert-manager.example.com                     True    
default              default.example.com                          True    
default              route-e07293b3-f782-45ae-9a52-d5aea6b0b3ba   True

If it still does not work, please check the controller logs in knative-serving ns:

$ kubectl  -n knative-serving logs controller-675478f86d-qpxpw -f |grep invalid

You will find the error like:

{"severity":"INFO","timestamp":"2023-07-31T10:13:43.935713201Z","logger":"controller.event-broadcaster","caller":"record/event.go:285","message":"Event(v1.ObjectReference{Kind:\"Namespace\", Namespace:\"\", Name:\"local-path-storage\", UID:\"664528d4-ca09-477c-8308-de61dd1809a6\", APIVersion:\"v1\", ResourceVersion:\"264\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' failed to apply domain template {{if index .Labels \"serving.kserve.io/inferenceservice\" }}{{index .Labels \"serving.kserve.io/inferenceservice\" }}.{{ else }}{{ end }}{{ .Namespace }}.{{ .Domain }} to domain example.com and namespace local-path-storage: invalid DomainTemplate: local-path-storage.example.com","commit":"bbdef8e","knative.dev/pod":"controller-675478f86d-r89lw"}

nak3 commented 1 year ago

It might be better to separate the namespace-wildcard-cert's "domain-template" from the original one by introducing a new configuration.

knative / serving