Error deploying on GKE: Forbidden

[vizero1]

Hi, I have a GKE cluster (standard) and try to deploy origin-ca-issuer on it. But after all my steps I get the following error: "re-queuing item due to error processing" err="admission webhook \"webhook.cert-manager.io\" denied the request: status.conditions: Forbidden: user \"system:serviceaccount:cert-manager:cert-manager-helm-ad2b5fe3\" does not have permissions to set approved/denied conditions for issuer {prod-issuer OriginIssuer cert-manager.k8s.cloudflare.com}" logger="cert-manager.controller" key="default/backend-ingress-tls-12345-1"

I use pulumi to deploy cert-manager helm chart and origin-ca-issuer helm chart with some manual step. Will give in the following my steps: To create the cert-manager I use the python library pulumi_kubernetes_cert_manager: `def deploy_cert_manager(provider): cert_manager_ns = core.v1.Namespace("cert-manager", metadata={"name": "cert-manager"})

cert_manager_component = cert_manager.CertManager(
    "cert-manager",
    install_crds=True,
    helm_options=cert_manager.ReleaseArgs(
        namespace="cert-manager",
        skip_crds=True,
    ),
    opts=ResourceOptions(provider=provider, depends_on=[cert_manager_ns]),
)

return cert_manager_component`

After the installation I install the crds for clsteroriginissuer and originissuer:

    crd_url = "https://raw.githubusercontent.com/cloudflare/origin-ca-issuer/v0.9.0/deploy/crds/cert-manager.k8s.cloudflare.com_originissuers.yaml"  # For OriginIssuer
    crd_url_cluster = "https://raw.githubusercontent.com/cloudflare/origin-ca-issuer/v0.9.0/deploy/crds/cert-manager.k8s.cloudflare.com_clusteroriginissuers.yaml"  # For ClusterOriginIssuer

    # Apply the chosen CRD
    origin_ca_issuer_crd = apply_origin_ca_issuer_crd(k8s_provider, name="origin-ca-issuer-crd", depends_on=[cert_manager_component], crd_url=crd_url)
    origin_ca_clusterissuer_crd = apply_origin_ca_issuer_crd(k8s_provider, name="origin-ca-clusterissuer-crd", depends_on=[cert_manager_component],
                                                      crd_url=crd_url_cluster)

def apply_origin_ca_issuer_crd(provider, name, depends_on, crd_url):
    origin_ca_crd = yaml.ConfigFile(
        name,
        file=crd_url,
        opts=ResourceOptions(provider=provider, depends_on=depends_on)
    )
    return origin_ca_crd

Then I install the origin-ca-issuer helm chart:

def deploy_origin_ca_issuer(provider, depends_on):
    origin_ca_issuer_namespace = core.v1.Namespace(
        "origin-ca-issuer-ns",
        metadata={"name": "origin-ca-issuer"},
        opts=ResourceOptions(provider=provider, depends_on=depends_on)
    )

    origin_ca_issuer_chart = Chart(
        "origin-ca-issuer",
        chart="oci://ghcr.io/cloudflare/origin-ca-issuer-charts/origin-ca-issuer",
        version="0.5.7",
        namespace="origin-ca-issuer",
        values={},
        opts=ResourceOptions(provider=provider, depends_on=[origin_ca_issuer_namespace] + depends_on)
    )

    return origin_ca_issuer_chart

Then I create the originissuer as written in the documentation:

kubectl create secret generic \
    --dry-run \
    -n default service-key \
    --from-literal key=v1.0-FFFFFFF-FFFFFFFF -oyaml

Save the output of this command into service-key.yaml and run following command: kubectl apply -f service-key.yaml -f deploy/example/issuer.yaml

When I check the prod-issuer I get following log:

origin-ca-issuer % kubectl get originissuer.cert-manager.k8s.cloudflare.com prod-issuer -o json | jq .status.conditions
[
  {
    "lastTransitionTime": "2024-07-30T16:42:12Z",
    "message": "OriginIssuer verified and ready to sign certificates",
    "reason": "Verified",
    "status": "True",
    "type": "Ready"
  }
]

Then I deploy my backend, service and ingress with pulumi:

def create_ingress(provider, namespace_name, name, host, service_name, service_port):
    return networking.v1.Ingress(
        name,
        metadata=ObjectMetaArgs(
            namespace=namespace_name,
            annotations={
                "cert-manager.io/issuer": "prod-issuer",
                "cert-manager.io/issuer-kind": "OriginIssuer",
                "cert-manager.io/issuer-group": "cert-manager.k8s.cloudflare.com",
                "external-dns.alpha.kubernetes.io/hostname": host
            },
        ),
        spec=networking.v1.IngressSpecArgs(
            rules=[
                networking.v1.IngressRuleArgs(
                    host=host,
                    http=networking.v1.HTTPIngressRuleValueArgs(
                        paths=[
                            networking.v1.HTTPIngressPathArgs(
                                path="/",
                                path_type="Prefix",
                                backend=networking.v1.IngressBackendArgs(
                                    service=networking.v1.IngressServiceBackendArgs(
                                        name=service_name,
                                        port=networking.v1.ServiceBackendPortArgs(number=service_port)
                                    )
                                )
                            )
                        ]
                    )
                )
            ],
            tls=[
                networking.v1.IngressTLSArgs(
                    hosts=[host],
                    secret_name=f"{name}-tls-12345"
                )
            ]
        ),
        opts=ResourceOptions(provider=provider),
    )

My pod and service are starting but with the ingress I have problems: Error syncing to GCP: error running load balancer syncing routine: error initializing translator env: secrets "backend-ingress-tls-12345" not found

In the cert-manager container I can find following logs: Setting lastTransitionTime for CertificateRequest "backend-ingress-tls-12345-1" condition "Approved" to 2024-07-30 16:41:26.176465605 +0000 UTC m=+93.442562707

"re-queuing item due to error processing" err="admission webhook \"webhook.cert-manager.io\" denied the request: status.conditions: Forbidden: user \"system:serviceaccount:cert-manager:cert-manager-helm-32108e2d\" does not have permissions to set approved/denied conditions for issuer {prod-issuer OriginIssuer cert-manager.k8s.cloudflare.com}" logger="cert-manager.controller" key="default/backend-ingress-tls-12345-1"

"certificate resource is not owned by this object. refusing to update non-owned certificate resource for object" logger="cert-manager.controller.ingress-shim" resource_name="backend-ingress-e9afea6e" resource_namespace="default" resource_kind="" resource_version="" related_resource_name="backend-ingress-tls-12345" related_resource_namespace="default" related_resource_kind="Certificate" related_resource_version="v1"

In the cert-manager-webhook container I find following log: http: TLS handshake error from XX.XXX.XXX:36738: EOF

In the origin-ca-issuer the logs are:

Guess the crd kind error were happening during the installation and later it worked but not sure on that. Any of you know what my error is? I did it as the README has documented but still I have no luck with that. Woul appreciate help. Thanks :)

[terinjokes]

Please ensure you've deployed the equivalent of the signer RBAC role: https://github.com/cloudflare/origin-ca-issuer/blob/7b16c1daa5d0dcd0c376eaefa47ce430f14be455/deploy/rbac/role-approver.yaml

[vizero1]

Is this not done with the helm chart? In general when I install the helm chart, do I need to run following commands: kubectl apply -f deploy/rbac kubectl apply -f deploy/manifests

The command kubectl apply -f deploy/crds I need to run as that is written in the Readme

[terinjokes]

It is part of the helm chart: https://github.com/cloudflare/origin-ca-issuer/blob/7b16c1daa5d0dcd0c376eaefa47ce430f14be455/deploy/charts/origin-ca-issuer/templates/issuer-clusterrole.yaml#L33. however the error suggests it's not been applied or the rolebinding is for the wrong service account.

[vizero1]

Applied it now manually but still the same error. Just for documentation if the steps are correct or I am missing something: 1.) Install cert-manager via helm chart (will be create in the cert-manager namespace) 2.) kubectl apply -f https://raw.githubusercontent.com/cloudflare/origin-ca-issuer/v0.9.0/deploy/crds/cert-manager.k8s.cloudflare.com_originissuers.yaml kubectl apply -f https://raw.githubusercontent.com/cloudflare/origin-ca-issuer/v0.9.0/deploy/crds/cert-manager.k8s.cloudflare.com_clusteroriginissuers.yaml

3.) Install 0.5.7 helm chart of origin-ca-issuer in namespace origin-ca-issuer 4.) Create secret with command:

kubectl create secret generic \
    --dry-run \
    -n default service-key \
    --from-literal key=v1.0-FFFFFFF-FFFFFFFF -oyaml

Save output into service-key.yaml file and run command: kubectl apply -f service-key.yaml -f deploy/example/issuer.yaml

This will create prod-issuer in namespace default And this prod-issuer I can then reference in my ingress.

Are this steps correct or am I missing something crucial?

[terinjokes]

Please make sure the service account name matches what you've deployed in cert-manager: https://github.com/cloudflare/origin-ca-issuer/blob/7b16c1daa5d0dcd0c376eaefa47ce430f14be455/deploy/charts/origin-ca-issuer/values.yaml#L99-L101

From the error it looks like you've called the service account "cert-manager-helm-ad2b5fe3"

[vizero1]

Thanks. That fixed it