"managed-certificate-role" is forbidden

[eric-basley]

Cannot deploy controller due to following error:

$ kubectl apply -f managed-certificate-controller.yaml

serviceaccount "managed-certificate-account" created clusterrolebinding.rbac.authorization.k8s.io "managed-certificate-binding" created deployment.apps "managed-certificate-controller" created Error from server (Forbidden): error when creating "managed-certificate-controller-old.yaml": clusterroles.rbac.authorization.k8s.io "managed-certificate-role" is forbidden: attempt to grant extra privileges: [{[] [gke.googleapis.com] [managedcertificates] [] []} {[] [] [configmaps] [] []} {[] [] [events] [] []}] user=&{eric.basley@redpelicans.com [system:authenticated] map[user-assertion.cloud.google.com:[APTNk9TTsJ4paIpwW7+/0xgKOISypeb+QUmyw9+4nzABDyxYrW+nnCS+kSKxWi0+dPy65pNZUX2scM0nDAjMd1hgcyrtKFPxasWi1a+DEO D9pslJgXAdTcNMS1d0/vqsZd8jTKBUCmUVy7MZl+vy6TR4eJkykvM/rcZnBjXj70IGbStbCE5GIc5ceg4fiiqhzPAZYUQitMTzTckytgGtNgCVBL3LSwr0jN8dX2wn4jNqTZyItA==]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/ /apis /apis/ /healthz /openapi /openapi/ /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/* /version /version/]}] ruleResolutionErrors=[clusterroles.rbac.authorization.k8s.io "managed-certificate-role" not found]

If I continue and create certificate:

$ kubectl describe mcrt/my-certificate

Warning BackendError 7m (x24 over 16m) managed-certificate-controller googleapi: Error 403: Insufficient Permission, insufficientPermissions

Thx for your help

[krzykwas]

Sorry, the README file needs updating. I tried to make some changes to the way this feature would work in open source Kubernetes (https://github.com/kubernetes/kubernetes/pull/71568), but I had to change the way in which I communicate with Ingress instead, that is why the README file rewrite got delayed.

Anyway - you need to enable compute-rw Cloud api access on the node on which you deploy the controller and then follow the advice from https://github.com/coreos/prometheus-operator/issues/357 „So in order to proceed without error, cluster-admin role should be added to current executing user, eg: kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin --user=your.google.cloud.email@example.org”

[kribor]

Since I'm sure more people will get stuck here, here are some things to help troubleshooting: troubleshooting logs are not common practice stdout = viewable in stackdriver, instead they are in /var/log/managed_certificate_controller.log

I'm by no means an expert on kubernetes role bindings but the suggestion to give myself cluster admin did not work. it left me with:

I0130 14:03:31.815933       6 reflector.go:169] Listing and watching *v1alpha1.ManagedCertificate from github.com/GoogleCloudPlatform/gke-managed-certs/pkg/clientgen/informers/externalversions/factory.go:117
E0130 14:03:31.817494       6 reflector.go:134] github.com/GoogleCloudPlatform/gke-managed-certs/pkg/clientgen/informers/externalversions/factory.go:117: Failed to list *v1alpha1.ManagedCertificate: managedcertificates.gke.googleapis.com is forbidden: User "system:serviceaccount:devops:managed-certificate-account" cannot list managedcertificates.gke.googleapis.com at the cluster scope
I0130 14:03:32.269274       6 reflector.go:169] Listing and watching *v1beta1.Ingress from k8s.io/client-go/informers/factory.go:131
E0130 14:03:32.270486       6 reflector.go:134] k8s.io/client-go/informers/factory.go:131: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:devops:managed-certificate-account" cannot list ingresses.extensions at the cluster scope

So I ran: kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin --user=system:serviceaccount:devops:managed-certificate-account

Then certificate was created and annotations added to ingress - but https still didn't work until I reapplied the ingress (could have been broken I suppose)

[kribor]

I made a complete guide, might help someone: https://github.com/kribor/gke-managed-certs-example

Update 2019-04-29: updated guide with custom service account via json in kubernetes secret and GOOGLE_APPLICATION_CREDENTIALS

[wbyoung]

@kribor if you want to update your guide, the suggestions from @bmhatfield in #14 about using GOOGLE_APPLICATION_CREDENTIALS is a bit safer. I just got this working and documented it here. I think it'd be great if your guide followed the same, more secure, setup.

[krzykwas]

I updated README and opened a new issue which lists guides by @wbyoung and @kribor. I will now close this one. Thanks for contributions!