search

hpi-schul-cloud / infra-otc-cert-manager-webhook

Cert manager acme dns01 webhook provider for the Open Telekom Cloud (OTC).
MIT License
7 stars 1 forks source link

ACME Challenge failed in OTC due forbidden resource creation otcdns #1

Open hcv-adaumann opened 1 year ago

hcv-adaumann commented 1 year ago

I followed your instructions but the ACME challenge failed: Error presenting challenge: otcdns.xxx-development.otc-cert-manager-webhook is forbidden: User "system:serviceaccount:xxx-certmanager:certmanager-cert-manager-controller" cannot create resource "otcdns" in API group "xxx-development.otc-cert-manager-webhook" at the cluster scope

I have no idea, that what I have done for troubleshooting:

(a) Setup: values.yaml

infra-otc-cert-manager-webhook:
  groupName: xxx-development.otc-cert-manager-webhook
  cert-manager:
    namespace: xxx-certmanager
    serviceAccountName: certmanager-cert-manager-webhook 
  image:
    repository: swr.eu-de.otc.t-systems.com/xxxxx-development/infra-otc-cert-manager-webhook
    tag: latest
    pullSecret: secretregistryotc

(b) Cluster issuer (Credentials secrets are existing too)

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory

    # Email address used for ACME registration
    email: info@xxxx.de # REPLACE THIS WITH YOUR EMAIL!!!

    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging-otcdms

    solvers:
      - dns01:
          webhook:
            groupName: xxx-development.otc-cert-manager-webhook
            solverName: otcdns
            config:
              authURL: "https://iam.eu-de.otc.t-systems.com:443/v3"
              region: "eu-de"

              # Only for local testing, if no secrets are available.
              # accessKey: ACCESSKEY
              # secretKey: SECRETKEY

              accessKeySecretRef:
                name: otcdns-credentials
                key: accessKey
              secretKeySecretRef:
                name: otcdns-credentials
                key: secretKey

(c) Ingress Configuration (Helm extracted)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: keycloak
  namespace: "xxx-keycloak"
  labels:
    app.kubernetes.io/name: keycloak
    helm.sh/chart: keycloak-15.1.7
    app.kubernetes.io/instance: keycloak
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: keycloak
  annotations:
    cert-manager.io/cluster-issuer: xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
    kubernetes.io/elb.class: performance
    kubernetes.io/elb.id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    kubernetes.io/elb.port: "443"
spec:
  ingressClassName: "cce"
  rules:
    - host: "xxx-cloud.de"
      http:
        paths:
          - path: /iam/
            pathType: ImplementationSpecific
            backend:
              service:
                name: keycloak
                port:
                  name: http
  tls:
    - hosts:
        - "xxxxxxxxxxxxxxxx.de"
      secretName: xxx.de-tls

(d) Certificates are created in differnt namepsace (not certmanager)

Name:         xxx-cloud.de-tls
Namespace:    xxx-keycloak
Labels:       app.kubernetes.io/component=keycloak
              app.kubernetes.io/instance=keycloak
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=keycloak
              helm.sh/chart=keycloak-15.1.7
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2023-08-24T10:39:49Z
  Generation:          1
  Owner References:
    API Version:           networking.k8s.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  keycloak
    UID:                   a0087ade-a18c-4988-aab8-21c638c04e08
  Resource Version:        4650231
  UID:                     fdd70cf8-c6ca-4b94-a420-0e0580c3cb5a
Spec:
  Dns Names:
    xxx-cloud.de
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
  Secret Name:  xxx-cloud.de-tls
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:        2023-08-24T10:39:49Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2023-08-24T10:39:49Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  xxx.de-tls-sj845
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    18m   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  18m   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "xxx-cloud.de-tls-sj845"
  Normal  Requested  18m   cert-manager-certificates-request-manager  Created new CertificateRequest resource "xxx-cloud.de-tls-7qdrq"

(e) Certificate request


Name:         xxx.de-tls-7qdrq
Namespace:    xxx-keycloak
Labels:       app.kubernetes.io/component=keycloak
              app.kubernetes.io/instance=keycloak
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=keycloak
              helm.sh/chart=keycloak-15.1.7
Annotations:  cert-manager.io/certificate-name: xxx-cloud.de-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: xxx-cloud.de-tls-sj845
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2023-08-24T10:39:49Z
  Generate Name:       xxx-cloud.de-tls-
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  xxx-cloud.de-tls
    UID:                   fdd70cf8-c6ca-4b94-a420-0e0580c3cb5a
  Resource Version:        4650252
  UID:                     5d016903-5319-4753-bfaf-9c5756121533
Spec:
  Extra:
    authentication.kubernetes.io/pod-name:
      certmanager-cert-manager-controller-5489f79646-7w4zj
    authentication.kubernetes.io/pod-uid:
      10ebd0e2-77fc-4ce1-ac98-69479264467a
  Groups:
    system:serviceaccounts
    system:serviceaccounts:xxx-certmanager
    system:authenticated
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2dqQ0NBV29DQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxsawp2bDd6dTlvdkozUU5URXoyRmxqS2NFZkswRFhqWmRYU3dMV3B2NXpQTGZnelRZZjkwOWdDNjYyWithWXVyWDE5CmhBYVJEemRnMW1SeENrdkxWT0JETGt5ZFBZWTZIZE1wY2VZMGEvWHVlOVE2RmlSYnpwaFBrbGdXVzFmMHlOQU4KLzZNcWwvM2ptSjl0WDJFaU9RbkVKRFF3RkZ0SzAxOGdzYlV1Y21VNXVXNDVDSEgzTFJ4VWNmRmFDYWVXSzBpcQpreVZ1VTdVdm9hRWdEanV2TVFuN3lkZ1RsV3ZEcll3QWVoVUVlZnp2eTM1QjdoVnlUQzVMVnYzMVV0Mys3RzNtCkFhSGdRVnVEcUdJeGZGM1A3UjN3b3MyTzJQVU1PMldFVGJFeHRWVFFEWXo1OHB5empwdFl6TVc2Um5JSm9rOWUKVXZDS1dLdGFkRVRzZ1JzUmFCVUNBd0VBQWFBOU1Ec0dDU3FHU0liM0RRRUpEakV1TUN3d0hRWURWUjBSQkJZdwpGSUlTYUdNdGRtbHphVzl1TFdOc2IzVmtMbVJsTUFzR0ExVWREd1FFQXdJRm9EQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBc3hqOHV4RVNQYjVaQ0ZxcW9nYlEwbjY1S2U5NmhnTytWeXlBNDNvZGxDTXcvekFRSEN5ZmtYR1AKZGNaSTVsNEdXSk5DeCtXVGprTmNNZUlSRFVyWnV1dVFBTWVVSi9nY1QxY2Roc0RKTFJFYzhUM3NrVHVYMENaSQpFdW1vaXAzK0VOcVR1Qjk3S3c4N1hpcmtpTTgvV2lqb3J5ejF4ay91d1RvM2xQNXJ0OU11NTJkc08zTHRBdUppCm1DYWZZNlFWekVjYWRkaWVIdklsL3VNaDVxNmJKYnlmVlFyRHl4RW5jc0ZJRmJXc3I2L1hjcjJQUzVCdUhXTTEKVVVuSGpPNWR0eUZBVFpIUjU3RGRRUENOSlVwRUcxdTg2K3JMeWxzVFNqQzVOTlprTVN2dWRpTGVRVkU3VVZjWApva21YQXkxNmJ6L0tyU1RWSjNYRUZ5aXU5R2sza0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
  UID:      b08953a3-459a-48e1-a43b-8e964fb5a6b1
  Usages:
    digital signature
    key encipherment
  Username:  system:serviceaccount:xxx-certmanager:certmanager-cert-manager-controller
Status:
  Conditions:
    Last Transition Time:  2023-08-24T10:39:49Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2023-08-24T10:39:49Z
    Message:               Waiting on certificate issuance from order xxx-keycloak/xxx-cloud.de-tls-7qdrq-3502903507: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason              Age   From                                                Message
  ----    ------              ----  ----                                                -------
  Normal  WaitingForApproval  24m   cert-manager-certificaterequests-issuer-ca          Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  24m   cert-manager-certificaterequests-issuer-venafi      Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  24m   cert-manager-certificaterequests-issuer-vault       Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  24m   cert-manager-certificaterequests-issuer-selfsigned  Not signing CertificateRequest until it is Approved
  Normal  WaitingForApproval  24m   cert-manager-certificaterequests-issuer-acme        Not signing CertificateRequest until it is Approved
  Normal  cert-manager.io     24m   cert-manager-certificaterequests-approver           Certificate request has been approved by cert-manager.io
  Normal  OrderCreated        24m   cert-manager-certificaterequests-issuer-acme        Created Order resource xxx-keycloak/xxx-cloud.de-tls-7qdrq-3502903507
  Normal  OrderPending        24m   cert-manager-certificaterequests-issuer-acme        Waiting on certificate issuance from order xxx-keycloak/xxx-cloud.de-tls-7qdrq-3502903507: ""

(f) Order

Name:         xxx-cloud.de-tls-7qdrq-3502903507
Namespace:    xxx-keycloak
Labels:       app.kubernetes.io/component=keycloak
              app.kubernetes.io/instance=keycloak
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=keycloak
              helm.sh/chart=keycloak-15.1.7
Annotations:  cert-manager.io/certificate-name: xxx-cloud.de-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: xxx-cloud.de-tls-sj845
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2023-08-24T10:39:49Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  xxx-cloud.de-tls-7qdrq
    UID:                   5d016903-5319-4753-bfaf-9c5756121533
  Resource Version:        4650254
  UID:                     7ce873d9-ad09-45e5-8b5d-4063b31bfcae
Spec:
  Dns Names:
    xxx-cloud.de
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2dqQ0NBV29DQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxsawp2bDd6dTlvdkozUU5URXoyRmxqS2NFZkswRFhqWmRYU3dMV3B2NXpQTGZnelRZZjkwOWdDNjYyWithWXVyWDE5CmhBYVJEemRnMW1SeENrdkxWT0JETGt5ZFBZWTZIZE1wY2VZMGEvWHVlOVE2RmlSYnpwaFBrbGdXVzFmMHlOQU4KLzZNcWwvM2ptSjl0WDJFaU9RbkVKRFF3RkZ0SzAxOGdzYlV1Y21VNXVXNDVDSEgzTFJ4VWNmRmFDYWVXSzBpcQpreVZ1VTdVdm9hRWdEanV2TVFuN3lkZ1RsV3ZEcll3QWVoVUVlZnp2eTM1QjdoVnlUQzVMVnYzMVV0Mys3RzNtCkFhSGdRVnVEcUdJeGZGM1A3UjN3b3MyTzJQVU1PMldFVGJFeHRWVFFEWXo1OHB5empwdFl6TVc2Um5JSm9rOWUKVXZDS1dLdGFkRVRzZ1JzUmFCVUNBd0VBQWFBOU1Ec0dDU3FHU0liM0RRRUpEakV1TUN3d0hRWURWUjBSQkJZdwpGSUlTYUdNdGRtbHphVzl1TFdOc2IzVmtMbVJsTUFzR0ExVWREd1FFQXdJRm9EQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBc3hqOHV4RVNQYjVaQ0ZxcW9nYlEwbjY1S2U5NmhnTytWeXlBNDNvZGxDTXcvekFRSEN5ZmtYR1AKZGNaSTVsNEdXSk5DeCtXVGprTmNNZUlSRFVyWnV1dVFBTWVVSi9nY1QxY2Roc0RKTFJFYzhUM3NrVHVYMENaSQpFdW1vaXAzK0VOcVR1Qjk3S3c4N1hpcmtpTTgvV2lqb3J5ejF4ay91d1RvM2xQNXJ0OU11NTJkc08zTHRBdUppCm1DYWZZNlFWekVjYWRkaWVIdklsL3VNaDVxNmJKYnlmVlFyRHl4RW5jc0ZJRmJXc3I2L1hjcjJQUzVCdUhXTTEKVVVuSGpPNWR0eUZBVFpIUjU3RGRRUENOSlVwRUcxdTg2K3JMeWxzVFNqQzVOTlprTVN2dWRpTGVRVkU3VVZjWApva21YQXkxNmJ6L0tyU1RWSjNYRUZ5aXU5R2sza0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
Status:
  Authorizations:
    Challenges:
      Token:        r6yVH_-cSlegASlsmRlVp7rZGGwPTBF0G9o_ivltR-4
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/7898030084/ShWR4A
      Token:        r6yVH_-cSlegASlsmRlVp7rZGGwPTBF0G9o_ivltR-4
      Type:         dns-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/7898030084/3dlcFw
      Token:        r6yVH_-cSlegASlsmRlVp7rZGGwPTBF0G9o_ivltR-4
      Type:         tls-alpn-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/7898030084/i4oMiA
    Identifier:     xxx-cloud.de
    Initial State:  pending
    URL:            https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7898030084
    Wildcard:       false
  Finalize URL:     https://acme-staging-v02.api.letsencrypt.org/acme/finalize/115937054/10473000664
  State:            pending
  URL:              https://acme-staging-v02.api.letsencrypt.org/acme/order/115937054/10473000664
Events:
  Type    Reason   Age   From                 Message
  ----    ------   ----  ----                 -------
  Normal  Created  25m   cert-manager-orders  Created Challenge resource "xxx-cloud.de-tls-7qdrq-3502903507-2917238827" for domain "xxx-cloud.de"

(g) ACME Challenge

Name:         xxx-cloud.de-tls-7qdrq-3502903507-2917238827
Namespace:    xxx-keycloak
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2023-08-24T10:39:51Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  xxx-cloud.de-tls-7qdrq-3502903507
    UID:                   7ce873d9-ad09-45e5-8b5d-4063b31bfcae
  Resource Version:        4650268
  UID:                     780aefbc-edff-48cd-bbd4-1c69c707562a
Spec:
  Authorization URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/7898030084
  Dns Name:           xxx-cloud.de
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   xxx-certmanager-clusterissuer-letsencrypt-staging-otcdns
  Key:      cGlSqs15z01PWk_PhWLi5WS4zm1QgQ4LnMs5vHmsenI
  Solver:
    dns01:
      Webhook:
        Config:
          Access Key Secret Ref:
            Key:     accessKey
            Name:    otcdns-credentials
          Auth URL:  https://iam.eu-de.otc.t-systems.com:443/v3
          Region:    eu-de
          Secret Key Secret Ref:
            Key:      secretKey
            Name:     otcdns-credentials
        Group Name:   xxx-development.otc-cert-manager-webhook
        Solver Name:  otcdns
  Token:              r6yVH_-cSlegASlsmRlVp7rZGGwPTBF0G9o_ivltR-4
  Type:               DNS-01
  URL:                https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/7898030084/3dlcFw
  Wildcard:           false
Status:
  Presented:   false
  Processing:  true
  Reason:      otcdns.xxx-development.otc-cert-manager-webhook is forbidden: User "system:serviceaccount:xxx-certmanager:certmanager-cert-manager-controller" cannot create resource "otcdns" in API group "xxx-development.otc-cert-manager-webhook" at the cluster scope
  State:       pending
Events:
  Type     Reason        Age                  From                     Message
  ----     ------        ----                 ----                     -------
  Normal   Started       27m                  cert-manager-challenges  Challenge scheduled for processing
  Warning  PresentError  6m20s (x9 over 27m)  cert-manager-challenges  Error presenting challenge: otcdns.xxx-development.otc-cert-manager-webhook is forbidden: User "system:serviceaccount:xxx-certmanager:certmanager-cert-manager-controller" cannot create resource "otcdns" in API group "xxx-development.otc-cert-manager-webhook" at the cluster scope
hcv-adaumann commented 1 year ago

Same problem if using a issuer instead of a clusterissuer.

Crenshinibon commented 2 months ago

Likely a problem with access rights in OTC itself.