Failed to validate namespace existence: default

[JorritSalverda]

When trying to configure the plugin it fails with the following message:

Failed to validate namespace existence: default Please check plugin log for more detail.

The only thing I can find in the go.cd server logs is the following:

ERROR [128@MessageListener for ServerPingListener] JMSMessageListenerAdapter:77 - Exception thrown in message handling by listener com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener@329b4cb2
 java.lang.RuntimeException: Interaction with plugin with id 'cd.go.contrib.elasticagent.kubernetes' implementing 'elastic-agent' extension failed while requesting for 'cd.go.elastic-agent.server-ping'. Reason: [The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request cd.go.elastic-agent.server-ping']
    at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:57)
    at com.thoughtworks.go.plugin.access.elastic.v3.ElasticAgentExtensionV3.serverPing(ElasticAgentExtensionV3.java:111)
    at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.serverPing(ElasticAgentExtension.java:82)
    at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.serverPing(ElasticAgentPluginRegistry.java:50)
    at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:32)
    at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:22)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:73)
    at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:53)
    at java.lang.Thread.run(Thread.java:748)
 Caused by: java.lang.RuntimeException: The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to handle request cd.go.elastic-agent.server-ping'
    at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:55)
    ... 8 common frames omitted

I've set up RBAC role, binding and service account with the following template and use https://kubernetes.default.svc.cluster.local as cluster url. I've copied the service account token from /var/run/secrets/kubernetes.io/serviceaccount/token and ca certificate data from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: server-role
 namespace: ${NAMESPACE}
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - get
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - list
  - get
- apiGroups:
  - ""
  resources:
  - pods
  - pods/logs
  verbs:
  - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: server-role-binding
  namespace: ${NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: server-role
subjects:
- namespace: ${NAMESPACE}
  kind: ServiceAccount
  name: server
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: server
 namespace: ${NAMESPACE}

Is there any other place to find the actual logs from the plugin?

[varshavaradarajan]

@JorritSalverda - for events and namespaces, the privileges must be cluster scoped. It should be a cluster role and a cluster role binding. Can you please try that out? Check https://github.com/kubernetes/charts/blob/master/stable/gocd/templates/gocd-ea-cluster-role.yaml and https://github.com/kubernetes/charts/blob/master/stable/gocd/templates/gocd-ea-cluster-role-binding.yaml for reference.

[varshavaradarajan]

Can you please clarify from which container you're copying the value for /var/run/secrets/kubernetes.io/serviceaccount/token ? And are you sure that the token matches the configured service account's token? You can match the values by doing

secret_name=$(kubectl get serviceaccount server --namespace=NAMESPACE  -o jsonpath="{.secrets[0].name}")
kubectl get secret $secret_name --namespace=NAMESPACE -o jsonpath="{.data['token']}" | base64 --decode

and

cat /var/run/secrets/kubernetes.io/serviceaccount/token inside the server container.

[JorritSalverda]

Changing the Role and RoleBinding to ClusterRole and ClusterRoleBinding seemed to have helped. I also had to remove a % from the end of the /var/run/secrets/kubernetes.io/serviceaccount/token file content. When using kubectl describe secret TOKEN_NAME as described in the form it doesn't show this % at the end, so that's the better way to find the value.

Hopefully the setup for go.cd server running in Kubernetes can be simplified by implementing https://github.com/gocd/kubernetes-elastic-agents/issues/44 for future users.