Can't transfer acl token from kubernetes

[chesnovskii]

I want to realize service mesh with consul connect. I have installed a consul cluster outside of kubernetes cluster with acl_enabled=true. I have consul agents running in kubernetes cluster. Consul agents have a consul token and they are registered successfully in consul. I have an application in kubernetes cluster which I want to register as service in consul. In kubernetes deployment.yaml I can transfer some consul params in annotations, such as:

• consul.hashicorp.com/connect-inject
• consul.hashicorp.com/connect-service-port
• consul.hashicorp.com/service-name

However, in documentation I can't find any parameters for transfer acl token to consul server, because of that I can't register an app service in consul.

How can I transfer consul acl token from kubernetes to consul server ?

[lkysow]

Hi Stanislav, Thanks for the issue. The way the connect injector works is that it adds an initContainer to each Pod. The initContainer registers the service (and its sidecar) with Consul. If using ACLs, it performs a consul login to retrieve an ACL token that allows it to register the service and sidecar.

The exact command is:

/bin/consul login -method="{{ .AuthMethod }}" \
  -bearer-token-file="/var/run/secrets/kubernetes.io/serviceaccount/token" \
  -token-sink-file="/consul/connect-inject/acl-token" \
  -meta="pod=${POD_NAMESPACE}/${POD_NAME}"

Where {{ .AuthMethod }} comes from the flag passed to the consul-k8s connect-inject command here: https://github.com/hashicorp/consul-helm/blob/57f9b8554426142692eda29afd2dab71d27fd564/templates/connect-inject-deployment.yaml#L55-L57

So to use injection with externally configured servers you need to:

1. Create a Kubernetes Auth Method. Note that your Consul servers will need to be able to talk to the Kubernetes API servers in order to access the Token Review API.
2. Use the name of the auth method you create in the connect-inject-deployment.yaml file I linked above. NOTE: we will need to make this configurable via the helm chart. Right now you'd have to clone the chart and add that yourself directly.
3. Enable the connect injector deployment: https://www.consul.io/docs/platform/k8s/connect.html#installation-and-configuration

The code for creating a Kubernetes Auth method is here: https://github.com/hashicorp/consul-k8s/blob/master/subcommand/server-acl-init/command.go#L598. I will try to write out the commands that you could run to do it yourself.

consul acl auth-method create \
  -name=<your name> \
  -type=kubernetes \
  -kubernetes-host=<string> \
  -kubernetes-ca-cert=<string> \
  -kubernetes-service-account-jwt=<string>

• -kubernetes-host is an address that routes to your Kube API server from your Consul servers
• -kubernetes-ca-cert is the ca.crt key of the Secret of a ServiceAccount that has the following RBAC rules: https://github.com/hashicorp/consul-helm/blob/master/templates/connect-inject-clusterrole.yaml#L12-L19
• -kubernetes-service-account-jwt is the token key of the above Secret
• See https://www.consul.io/docs/acl/auth-methods/kubernetes.html and https://www.consul.io/docs/commands/acl/auth-method/create.html for more details.

consul acl binding-rule create \
  -method=<name of method from above> \
  # An expression for which types of service accounts are allowed to auth
  -selector="serviceaccount.name!=default" \
  -bind-type=service \
  -bind-name="${serviceaccount.name}"

• See https://www.consul.io/docs/commands/acl/binding-rule/create.html for more details.

When the auth-method and binding-rule are created, the consul login commands run by the initContainer should succeed.

We need to make the auth-method flag configurable and also document this so I'm going to leave the ticket open but please let us know if this works for you. I haven't yet tested this myself but wanted to write up what I think should work ASAP.

[hamishforbes]

I stumbled across the issue trying to figure out why my Connect proxy service wasn't working. Using a combination of @lkysow's comment above, the Consul acl auth method docs and the chart templates I have got it working.

Few gotchas for anyone else who finds this issue:

The chart docs page doesn't mention the new connectInject.overrideAuthMethodName value yet. tl;dr: You don't have to match up the consul acl auth method name with the values expected by the chart anymore. Create the auth method with whatever name you like and then set that name in the values.

The example manifests on the auth method page don't include the ServiceAccount or any help on how to get the required JWT / CA / Hostname values (probably out of scope for that page).

You need something along the lines of

# RBAC configuration for consul external acl auth method
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: consul-auth-method-delegator-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: consul-auth-method
    namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: consul-auth-method-serviceaccount-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: consul-auth-method-role
subjects:
- kind: ServiceAccount
  name: consul-auth-method
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: consul-auth-method-role
  namespace: kube-system
rules:
- apiGroups: [""]
  resources:
    - serviceaccounts
  verbs:
    - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: consul-auth-method
  namespace: kube-system

From there you can find the required JWT and ca cert (Base64 encoded) from kubectl e,g. kubectl -n kube-system get secret (kubectl -n kube-system get sa/consul-auth-method -o'jsonpath={.secrets[0].name}') -o yaml

Most of the Consul on Kubernetes docs assume that you're starting from scratch with Consul, but there must be plenty of people in the same situation as me. With an existing (non-k8s) Consul deployment which we are adding k8s to, or just deploying additional k8s clusters into the same Consul environment.

Some more docs or guides like this one for integrating a k8s cluster with an existing Consul deployment would be fantastic

[antonyaugustus]

@hamishforbes can you shed more light on your kubernetes auth method. Do you need the JWT from the kube-system service account?

[antonyaugustus]

Hi,

Any one able to give me some suggestions. I have Kubernetes AuthMethod Type with the JWT for Kubernetes Service Account - hashicorp-consul-connect-injector-authmethod-svc-account. This SA is in consul-system namespace and so are the consul agent daemonsets and connect-injector pods. While the test demo pods are running on “test” namespace.

Policy Definition is pretty open for all service and service_prefix. service “” { policy = “write” intentions = “write” } service_prefix “” { policy = “write” intentions = “write” } service_prefix “static-server” { policy = “write” } service_prefix “static-client” { policy = “write” }

Bound the policy/role to AuthMethod with different selectors. I tried different combination but I assume there are better ways.

./consul acl binding-rule create -method=auth-method-consul-test -bind-type=role -bind-name=‘test-ns-role’ -selector=‘serviceaccount.namespace==“test”’ ./consul acl binding-rule create -method=auth-method-consul-test -bind-type=role -bind-name=‘test-ns-role’ -selector=‘serviceaccount.namespace==“consul-system”’ ./consul acl binding-rule create -method=auth-method-consul-test -bind-type=role -bind-name=‘test-ns-role’ -selector=‘serviceaccount.name==“static-server”’ ./consul acl binding-rule create -method=auth-method-consul-test -bind-type=role -bind-name=‘test-ns-role’ -selector=‘serviceaccount.name==“static-client”’

Below are some of the logs from different containers in static-client pod

kubectl logs -n test static-client consul-connect-inject-init Registered service: static-client-sidecar-proxy Registered service: static-client

Here is the result from trying to access the service from inside the client pod

root@static-client:/# curl -v http://127.0.0.1:1234

Rebuilt URL to: http://127.0.0.1:1234/ Hostname was NOT found in DNS cache Trying 127.0.0.1… Connected to 127.0.0.1 (127.0.0.1) port 1234 (#0) GET / HTTP/1.1 User-Agent: curl/7.35.0 Host: 127.0.0.1:1234 Accept: / Empty reply from server Connection #0 to host 127.0.0.1 left intact curl: (52) Empty reply from server It looks like the static server is listening

kubectl logs -n test static-server static-server 2020/03/26 18:19:51 Server is listening on :8080

Thank you again for you assistance.

[pratiklotia]

@lkysow Still pretty new to consul, so apologies for amateur comments/suggestions:

I'm trying to realize a similar environment that the author mentions here. My final goal is to have Mesh Gateways with Connect in a K8s client & bare-metal server environment. The solution to add ACL tokens for services running outside K8s is pretty simple and straightforward as described here.

In contrast, to do the same thing via K8s pods appears a much longer and complicated process. I'm going to try this out and see where I reach. Is there any scope to change the solution to simply adding the ACL token in the K8s object definition (yaml)?

Also, I'm curious, why haven't these details made its way to the official Consul documentation - its over 6 months since this issue was brought up? It took me a while to wrap my head around finding the right solution and landing here. If you need any help with documentation, I'm happy to offer any assistance.

Thank you.

[hamishforbes]

Hashicorp have recently published this new guide: https://learn.hashicorp.com/vault/getting-started-k8s/external-vault#install-the-vault-helm-chart-configured-to-address-an-external-vault

which includes a walkthrough on setting up consul <-> k8s auth 👍

[pratiklotia]

@hamishforbes Thank you for the link. On a quick glance, it looks like this guide is mainly for vault-k8s integration. I don't see any mention of consul. I will check the guide in detail to see if I can locate the consul stuff.

[hamishforbes]

Oops, my bad! I had much the same issues getting my existing Vault cluster integrated with k8s, must've got the issues mixed up...

[pratiklotia]

All good, my friend. Thank you.