Unknown user "system:serviceaccount:weave:weave-agent"

[leth]

https://sentry.io/weaveworks/launcher-agent/issues/443372991/

*errors.errorString: failed to retrieve deployment: deployments.extensions "weave-agent" is forbidden: User "system:serviceaccount:weave:weave-agent" cannot get deployments.extensions in the namespace "weave": clusterrole.rbac.authorization.k8s.io "weave-agent" not found
Unknown user "system:serviceaccount:weave:weave-agent"
  File "/go/src/github.com/weaveworks/launcher/agent/main.go", line 71, in logError
  File "/go/src/github.com/weaveworks/launcher/agent/main.go", line 84, in updateAgents

failed to retrieve deployment: deployments.extensions "weave-agent" is forbidden: User "system:serviceaccount:weave:weave-agent" cannot get deployments.extensions in the namespace "weave": clusterrole.rbac.authorization.k8s.io "weave-agent" not found
Unknown user "system:serviceaccount:weave:weave-agent"

[dlespiau]

Reproduced the error on a test GKE cluster.

This is the result of a failed onboarding process, when we fail to create a ClusterRole with the admin rights. To simulate the onboarding error, I removed gcloud from my PATH. This looks like the following:

serviceaccount "weave-agent" configured
clusterrolebinding "weave-agent" configured
deployment "weave-agent" created
Error from server (Forbidden): clusterroles.rbac.authorization.k8s.io "weave-agent" is forbidden: attempt to grant extra privileges:.

• The Deployment only refers to the service account, both service account and Deployment are successfully created. We get an actual agent pod on the cluster, with a service account and the cluster role that we try to associate with the service account hasn't bee created (that's the error above).
• the agent logs show the error in question:

  ...
  time="2018-02-26T18:56:42Z" level=info msg="Updating self from https://get.dev.weave.works/k8s/agent.yaml?instanceID=REDACTED"
  time="2018-02-26T18:56:42Z" level=error msg="Failed to fetch latest deployment replicateset revision: failed to retrieve deployment: deployments.extensions \"weave-agent\" is forbidden: User \"system:serviceaccount:weave:weave-agent\" cannot get deployments.extensions in the namespace \"weave\": clusterrole.rbac.authorization.k8s.io \"weave-agent\" not found\nUnknown user \"system:serviceaccount:weave:weave-agent\""

• This agent pod will try to self-update itself every hour, leading a high count of errors.

We could either:

• Try to clean up when the final kubectl errors out, given the apply isn't atomic
• Split the agent manifests in 2 groups. The first one will create service account, clusterrole, clusterrolebinding. The second one will only have the agent Deployment. We won't apply the Deployment if we fail applying the first manifests.