Kubernetes Target support

[saada]

Is your feature request related to a problem? Please describe.

Boundary provides an easy-to-use, platform-agnostic way to access all of your hosts and services across clouds, Kubernetes clusters, and on-premises datacenters through a single workflow based on trusted identity. It lets you remove hard-coded credentials and firewall rules, and makes access control more dynamic. ~ https://www.hashicorp.com/blog/hashicorp-boundary

As per the above announcement, Kubernetes clusters as targets is either an existing or planned feature. Is there any guidance on how to use Boundary today for Kubernetes use-cases? If not, happy to contribute it as a target.

Describe the solution you'd like I would like to login to any Kubernetes cluster that exists in a given Boundary project using an expiring service account for a limited amount of time with specified RBAC permissions to my user. I expect a new Kubernetes context to be added to either export KUBECONFIG= or to my ~/.kube/config file.

Describe alternatives you've considered Should I write my own target plugin?

Explain any additional use-cases Apart from temporary access to a k8s cluster, it would be interesting to have the ability to request different RBAC permissions. For example, maybe I just want read-only access, and I can escalate my access on a need basis.

Additional context Existing solutions focused on Kubernetes:

• kconnect

[mootpt]

@saada +1 to this. Was considering hacking on a Kubernete Auth secrets engine for Vault similar to Nomad secrets engine.

[jefferai]

@saada Thanks for the report!

This is something we're very interested in but we have not yet thought through the design. There are a few ways we could go:

1. Allow users to run a client in a "listening mode" where the address/port remain fixed but each incoming connection to a listener triggers a session authorization at that time -- allows anything connecting to that port to trigger authorization but may be perfectly fine in most cases.
2. Exec kube or kube-related binaries with specific configuration information in e.g. KUBECONFIG= via something like boundary connect kubernetes
3. Other things?

All this is to say -- don't start writing something just yet...we definitely want this but we'd like to nail down a design with interested parties that will be the best option for everyone.

[saada]

@mootpt let me know if you would like to collaborate on this. You can DM me 🐦

@jefferai , makes sense... In terms of implementation/design, I would recommend looking at kconnect (written in Go and can be used as an internal/external dependency)

Here's my thoughts on where the design could go.

CLI

boundary connect k8s <FLAGS>
--kaas=[static|eks|aks|gke|...] list of kubernetes as a service providers to choose from. Special 
 options are "static" and "all". "static" refers to a statically added cluster. "all" means list all clusters in all available providers. If omitted, the user is prompted to choose from a list of authorized options.
--region=[us-east-1|us-east-2] If omitted, the user is prompted to choose from a list of authorized options.
--cluster="<my-cluster-name>" If omitted, the user is prompted to choose from a list of authorized options.

Kubernetes as a Service (KaaS) Dynamic Discovery

After IDP auth, find accessible clusters in your project, and prompt user to pick which cluster they want to connect to, in which region, in which cloud provider.

Static Discovery

If "static" is chosen as the KaaS provider, a list of clusters show up that were added manually to Boundary and not discovered via cloud provider APIs; such as AWS GO SDK.

Auth

Once IDP authentication is established, Boundary takes care of talking to Vault, ask for a temporary service account for a given role.

# setup vault with a new cluster
vault secrets enable kubernetes

# add static cluster provider
vault write kubernetes/roles/on-prem-clusters \
    plugin_name=kubernetes-secrets-plugin \
    static_provider="$HOME/.kube/config" \
    allowed_contexts="my-on-prem-cluster-.*" \ # multiple clusters are listed in the same kubeconfig
    policy_document=-<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  # "namespace" omitted since ClusterRoles are not namespaced
  name: secret-reader
rules:
- apiGroups: [""]
  #
  # at the HTTP level, the name of the resource for accessing Secret
  # objects is "secrets"
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
EOF

# add dynamic cluster provider
vault write kubernetes/roles/eks-dev-clusters \
    plugin_name=kubernetes-secrets-plugin \
    dynamic_provider=aws/creds/eks-dev-clusters \
    allowed_clusters="dev-.*" \
    default_ttl="1h" \
    max_ttl="24h" \
    policy_document=-<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  # "namespace" omitted since ClusterRoles are not namespaced
  name: secret-reader
rules:
- apiGroups: [""]
  #
  # at the HTTP level, the name of the resource for accessing Secret
  # objects is "secrets"
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
EOF

# list accessible clusters
vault read kubernetes/creds/on-prem-clusters # returns a list of accessible clusters that boundary can parse
vault read kubernetes/creds/eks-dev-clusters # returns a list of accessible clusters that boundary can parse

# get temporary kubeconfig to access a specific cluster
vault read kubernetes/creds/eks-dev-clusters/dev-1
vault read kubernetes/creds/on-prem-clusters/my-on-prem-cluster-1

Boundary can then call the above read endpoints from vault and create an intractable experience for users.

[jefferai]

Thanks for the writeup and pointer to kconnect! Going to unpack a few things here:

First, I think it would be useful to draw a distinction between a target, a host catalog, and boundary connect.

boundary connect is authorizing a session against a target and passing traffic locally. So you can run kconnect and pass it the IP/port of a local session (or have boundary connect run it via -exec or a subcommand) and have kconnect do everything it does now via a normal TCP session.

A target contains the information necessary to make a connection from a Boundary worker to the end system. Pretty much any other target that we create would be doing some kind of useful extra work that we are purposefully hiding from the client (e.g. to inject credentials that the client never sees -- Vault integration is on our very near roadmap so that we can support exactly this).

A host catalog is something that can connect to and discover hosts and/or services. Right now there's just static but it can be written to support all kinds of things. For instance, a Consul host catalog could ingress services from the Consul catalog and expose each service as a host set that you can use in a target (this kind of workflow is a key reason why targets operate on host sets instead of hosts, even though for static this makes things more onerous). So there are two possibilities here: one is a host catalog that discovers kube clusters from some central service and handles creating the host sets to point targets to them. Another is a host catalog that connects to a kube cluster and exposes the pods inside as host sets. Both are totally valid and both are things we've been thinking about!

Why I wanted to lay all that out is that your instincts are exactly correct in terms of what you want from an end to end perspective, and exactly what we want too! But the actual implementation would span across different parts of Boundary. Letting the client pick all of the options about which cluster to connect to, a la kconnect, would do an end-run around Boundary's core purpose of ensuring that a particular connection is authorized against specific host(s) and tracked and can be canceled. Doing this the Boundary way requires the discovery to offer up to a target/targets available endpoints via host sets; the target to be configured to do [stuff]; and the client to then make it as seamless as possible for the end user while still going through a session authorized by that target.

Regarding the [stuff] there are even a lot of options there. A kube target could act mostly as a TCP proxy but inject credentials; or it could protocol decode the underlying grpc stream and enable a lot more functionality. Arguably the former can be handled sufficiently with the tcp target by having Boundary fetch just-in-time dynamic credentials for kube at session authorization time and return it to the user, with Boundary then revoking those credentials in Vault when the session is terminated. That's the easier and faster way for this to be supported, but it also makes it more complicated for the client. Having Boundary inject credentials likely means protocol decoding grpc, which is a longer and more in-depth endeavor but pays off in many ways.

That all said, not all of this needs to be implemented at the same time. For instance, a host catalog that can discover Kube clusters from cloud services is independent from having a kube-specific target type (although I think a question that would come up is whether it would be independent from e.g. an aws ec2 host catalog, or whether an aws catalog could support both styles of discovery without becoming a monster to configure and maintain).

I truly hope I'm not dampening your enthusiasm here, it'd be great if there was a super quick solution, but separating these concerns and ensuring that we're putting the right pieces in the right abstractions pays off in terms of ensuring that Boundary is doing what it promises to do. Happy to hear your thoughts on all of the above.

[saada]

Makes sense. Are there guidelines for writing custom host catalogs? For example... Would it be possible for a host catalog to get aws credentials from Vault to fetch a list of EKS clusters and expose those as dynamic hosts?

[jefferai]

There aren't guidelines yet, no. It's just so early :-) You could look at the static host catalog for some initial ideas. But one thing worth cluing you into is how we've been thinking about host catalogs, which I'll do in two examples:

1. Consul: Every service maps to a host set; the hosts in the set are populated from the nodes in Consul that are implementing that service. The default port for the service is an aspect of the host set, but each host can specify its own port (because Consul lets you do this so we need to support it; this also necessarily means that every service on every host actually maps to a different host ID within the catalog). This isn't really implemented this way yet because we haven't needed to but the idea is that the target goes to the host sets and says "give me a list of hosts and associated addresses". Then it's up to the host set implementation to provide that information. So the Consul host set would give a list of addresses, substitute in the service port if any hosts specifically don't have a port set, and if no port comes back, we'd use the target's default port.

2. AWS EC2: We query EC2 for a list of instances and store them and their attributes as hosts in the database. Host sets are comprised, essentially, of a set of query parameters. It could be as simple as a particular instance ID to indicate a single instance, a set of ORd instance IDs, or it could be something like "all instances with this AMI ID in this VPC". This allows the host sets to be populated dynamically every time we refresh from EC2 and keeps you from having to manually manage the sets.

As you can see, quite different approaches, but the host catalog/host set/host abstraction is one that can work for those different sources. A question for you to think about, then, is: what would building a host set look like from the information that you queried?

Regarding credentials: I'm not sure we'd want to build in Vault logic into every single host catalog. I think a preferred approach would be to use Vault Agent's templating capabilities to write out (e.g. to ramdisk) a [credentials file]. Then we could just use the normal AWS SDK's credential capabilities.

[HeshamAboElMagd]

@jefferai I know that is a bit unrelated but is there a way to deploy boundary itself within kubernetes cluster? helm chart or some sort of a thing ?

[denibertovic]

Hey @jefferai is this still in the works? The reason I ask is because this article makes it seems like it's already implemented but I can't seem to find any docs about it and the reference/demo deployment listed in the article does not showcase this.

[jefferai]

We don't have a kubernetes target, no. But we have a connect helper for the 'boundary connect' command to make it easy to use the tcp target. See 'boundary connect kube' for info.

[springroll12]

@jefferai

I see that there is a style option available to the kube helper, but I don't see how this addresses the kubeconfig management that @saada suggested:

I would like to login to any Kubernetes cluster that exists in a given Boundary project using an expiring service account for a limited amount of time with specified RBAC permissions to my user. I expect a new Kubernetes context to be added to either export KUBECONFIG= or to my ~/.kube/config file.

Is there any guidance on how to achieve this with boundary or is this sort of thing completely out of scope and left to the post-authenticated tcp connection to figure out?

A kube target could act mostly as a TCP proxy but inject credentials; or it could protocol decode the underlying grpc stream and enable a lot more functionality. Arguably the former can be handled sufficiently with the tcp target by having Boundary fetch just-in-time dynamic credentials for kube at session authorization time and return it to the user, with Boundary then revoking those credentials in Vault when the session is terminated.

This seems promising, especially given the example given by @saada. Is this essentially what you're suggesting boundary users do for now until the grpc decoding work is complete?

[saada]

All this is to say -- don't start writing something just yet...we definitely want this but we'd like to nail down a design with interested parties that will be the best option for everyone.

@jefferai , any updates on this design with interested parties? Happy to join these design discussions if it helps.

[jefferai]

@saada Nobody took me up on that so no discussions happened. At this point the team is fairly slammed and I doubt anything more integrated with Kube than the existing style for boundary connect that helps with the TLS management is going to happen in the next couple of months. Additionally I think integrating with Kubernetes catalogs are a significant ways off, in no small part because we've been thinking about plugins in the intermediate time and we will need to sort out where we will support plugins, what the interfaces will look like, etc; and we may want all host catalogs that aren't static to be plugin-based.

That said, some Vault integration for credential brokering is coming soon. The kube helper for boundary connect could then use credentials provided from Vault when the session is authorized and inject them into a Kubernetes call. I think. I'm not super duper familiar with the intricacies of kubectl at this point. :-)

[alexellis]

Hi folks :wave:

Regarding this comment:

https://github.com/hashicorp/boundary/issues/684#issuecomment-709275570

I was curious how you were planning on populating an endpoint in the Kubeconfig file under the server: field.

Unless TLS insecure is used, then whatever local port or DNS entry the service kubernetes.default.svc is mapped to, it won't match the certificate that the API server has and managed solutions like GKE and EKS have no way to update the TLS SAN names to add in the endpoint that will be used to connect over the tunnel / proxy.

Case in point would be this article by @jsiebens] where he forwards several Kubernetes API servers (using K3s) into a central cluster using tunnels, each has a ClusterIP and internal DNS name, but the cert won't match. Johan then updates K3s to give it an extra TLS SAN name. The same can be done for kubeadm - just not with a managed cloud.

This makes me wonder how other tools like Teleport handle the scenario, because they support this use-case also.

I'd love to hear your thoughts here. Does my explanation make sense?

[charandas]

Teleport runs an agent in each of the clusters and the end-user usesteleport commands tsh kube login <cluster_name> to set up their kubeconfig. It proxies all the traffic from the central teleport server, so I doubt it will run into the SAN name issue you point above @alexellis.

[denibertovic]

That said, some Vault integration for credential brokering is coming soon. The kube helper for boundary connect could then use credentials provided from Vault when the session is authorized and inject them into a Kubernetes call. I think. I'm not super duper familiar with the intricacies of kubectl at this point. :-)

@jefferai is this vault integration you speak of released? I saw this so I imagine it exists but I didn't find any mention of kubernetes so perhaps that part is still in the works?

[springroll12]

With the release of v0.7 it seems the plugin system is taking shape. Is there an update on Kubernetes target support, or at least how the community might build a plugin to support such targets?

[yongzhang]

any good news on this?

[anoncam]

@jefferai I know that is a bit unrelated but is there a way to deploy boundary itself within kubernetes cluster? helm chart or some sort of a thing ?

so there is a janky one I have gotten working by doing some awful things. I think longterm for a myriad of important reasons, hashi should be owning the chart and standards for deployment, given the stage of the product tho i can share this chart

there are three init containers, the important one being an init job that does the initial db migrations and then there is an init container for the controllers and workers. There isn't a volume mount though supported in the chart (that I saw) so the security context breaks the deployment.

I had a lot of weirdness with the module. ran ran and ran for quite some time and made nothing in kubernetes.