Troubleshooting doc: Document missing cluster-admin privileges error/solution

[timuckun]

This may be because I have cert-manger installed in another cluster

INFO Creating non-existent clusterroles cert-manager ERROR Error: Error updating clusterroles cert-manager: clusterroles.rbac.authorization.k8s.io "cert-manager" is forbidden: attempt to grant extra privileges: [{[get] [certmanager.k8s.io] [certificates] [] []} {[list] [certmanager.k8s.io] [certificates] [] []} {[watch] [certmanager.k8s.io] [certificates] [] []} {[create] [certmanager.k8s.io] [certificates] [] []} {[patch] [certmanager.k8s.io] [certificates] [] []} {[update] [certmanager.k8s.io] [certificates] [] []} {[delete] [certmanager.k8s.io] [certificates] [] []} {[get] [certmanager.k8s.io] [issuers] [] []} {[list] [certmanager.k8s.io] [issuers] [] []} {[watch] [certmanager.k8s.io] [issuers] [] []} {[create] [certmanager.k8s.io] [issuers] [] []} {[patch] [certmanager.k8s.io] [issuers] [] []} {[update] [certmanager.k8s.io] [issuers] [] []} {[delete] [certmanager.k8s.io] [issuers] [] []} {[get] [certmanager.k8s.io] [clusterissuers] [] []} {[list] [certmanager.k8s.io] [clusterissuers] [] []} {[watch] [certmanager.k8s.io] [clusterissuers] [] []} {[create] [certmanager.k8s.io] [clusterissuers] [] []} {[patch] [certmanager.k8s.io] [clusterissuers] [] []} {[update] [certmanager.k8s.io] [clusterissuers] [] []} {[delete] [certmanager.k8s.io] [clusterissuers] [] []} {[get] [] [secrets] [] []} {[list] [] [secrets] [] []} {[watch] [] [secrets] [] []} {[create] [] [secrets] [] []} {[patch] [] [secrets] [] []} {[update] [] [secrets] [] []} {[delete] [] [secrets] [] []} {[get] [] [configmaps] [] []} {[list] [] [configmaps] [] []} {[watch] [] [configmaps] [] []} {[create] [] [configmaps] [] []} {[patch] [] [configmaps] [] []} {[update] [] [configmaps] [] []} {[delete] [] [configmaps] [] []} {[get] [] [services] [] []} {[list] [] [services] [] []} {[watch] [] [services] [] []} {[create] [] [services] [] []} {[patch] [] [services] [] []} {[update] [] [services] [] []} {[delete] [] [services] [] []} {[get] [] [pods] [] []} {[list] [] [pods] [] []} {[watch] [] [pods] [] []} {[create] [] [pods] [] []} {[patch] [] [pods] [] []} {[update] [] [pods] [] []} {[delete] [] [pods] [] []} {[get] [extensions] [ingresses] [] []} {[list] [extensions] [ingresses] [] []} {[watch] [extensions] [ingresses] [] []} {[create] [extensions] [ingresses] [] []} {[patch] [extensions] [ingresses] [] []} {[update] [extensions] [ingresses] [] []} {[delete] [extensions] [ingresses] [] []} {[create] [] [events] [] []} {[patch] [] [events] [] []} {[update] [] [events] [] []}] user=&{tim.uckun@csst.co.nz [system:authenticated] map[user-assertion.cloud.google.com:[............]]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews selfsubjectrulesreviews] [] []} {[get] [] [] [] [/api /api/ /apis /apis/ /healthz /openapi /openapi/ /swagger-2.0.0.pb-v1 /swagger.json /swaggerapi /swaggerapi/ /version /version/]}] ruleResolutionErrors=[]

[anguslees]

Thanks for the report.

This is an error while creating a ClusterRole object within k8s, which is entirely local to the cluster - the other cluster shouldn't affect this particular operation. The error is "updating ... is forbidden: attempt to grant extra privileges" which is caused when trying to create a ClusterRole that grants privileges that the current k8s user does not already have.

GKE (unlike every other default k8s setup) does not give the initial user full "admin" privileges.

You need to run:

GCLOUD_USER=$(gcloud info --format='value(config.account)')
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin \
  --user=${GCLOUD_USER}

... and then retry the kubeprod install gke ... command that led to the above (from the same directory, with the previously generated kubeprod-autogen.json file). If you are specifying a non-default --config for kubeprod install then you will need to also ensure the above commands are acting on the user/cluster from the non-default kubernetes context.

Please confirm the above fixes it for you!

I expect this will be very common, since it is a very easy step to overlook in the GKE quickstart docs - so I have turned this into a doc improvement bug.

[timuckun]

I had already done that step as specified in the instructions.

I was able to get past this error by inserting the command

kubectl config use-context ${GCLOUD_K8S_CLUSTER}

before running all other commands and skipping the --context filter.

In fact I had to do that because it was installing it whatever context was set by ~/.kube/config. That might be a good thing to put in the docs.

[anguslees]

@timuckun oh! Yes, a lot of the quickstart assumes we're using the current/default k8s config file and context just for brevity. Thanks for the followup, I'll see if we can make the initial context setup more robust to avoid the next person stumbling into the same issue...

[timuckun]

So I am curious.

If I have a config file couldn't I skip all the other steps about setting up a oauth key and going throught the gcloud login portions of the instructions?

In other words isn't a kubectl config set-context sufficient?

[EamonKeane]

I was getting the above message too, the solution for me was to turn off using client certificate - gcloud config set container/use_client_certificate False

[anguslees]

If I have a config file couldn't I skip all the other steps about setting up a oauth key and going throught the gcloud login portions of the instructions?

In other words isn't a kubectl config set-context sufficient?

No. kubeprod install does two things: It a) sets up various non-kubernetes things on your cloud, and b) expands the jsonnet manifests and pushes the results to your cluster.

Yes, (b) just requires a functional kubectl config file (and defaulting to a user with appropriate cluster-admin access).

(a) however requires a functional gcloud config and access (assuming we're talking about GKE here), which will be used for (eg) creating a restricted service account used for DNS management. On GKE in particular, there is no API available to configure OAuth2 so this part must be done manually through the web interface (see GCP feature request)

---

I was getting the above message too, the solution for me was to turn off using client certificate - gcloud config set container/use_client_certificate False

Thanks. This is also surprising, since we read the "real" ~/.kube/config (or whatever value of KUBECONFIG or command-line flag), and should support client TLS certificate authentication just fine.. I will test this.

[EamonKeane]

thanks for the link to that GCP feature request... I have wondered for some time why you can't automate that. As good as GCP is (much better than alternatives) there are small things like this and still offering only postgres 9.6 that make me wonder who decides what to build internally at GCP.

[falfaro]

Does this issue still apply?