Ability To Send Custom Headers From kubectl

[joshkurz]

What would you like to be added?

It would be nice to have the option to pass along custom headers from kubectl to the k8s api. This would allow for more security options to be used, possibly tracing options or monitoring options as well.

Thinking it could look something like

kubectl get pods --customheader foo:bar --customheader far:bar 

It should work with any command as it would need to be global, since any api call could possibly need the custom header ability.

It could also be an option in the kubeconfig per cluster. That way it would not need to be set every single command.

Why is this needed?

It's a best practice to lock down the k8s api, via networking layer, and not run it on the public internet. This requires the api to either have a firewall on it that only allows traffic from specific IPs or to be internal to a network and then some proxy is required to reach it externally, which is typically locked down to IPs as well.

If we had the ability to pass custom headers, we could use zero trust security products that could front the api and then allow much stronger identity based access to the API. This would allow for multi factor auth on k8s API, by locking down the API to only users that need access to it, then on top of that, k8s based authn/z. Allowing for both factors of security is a good way to prevent zero day issues effecting your api. This also allows for k8s to be fully supported by zero trust solutions.

[k8s-ci-robot]

@joshkurz: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[brianpursley]

/sig cli

[sftim]

If client-go supports this, we could transfer this issue to k/kubectl.

[brianpursley]

@sftim Yes, this should be transferred to k/kubectl for better visibility for sig-cli.

/transfer kubectl

[seans3]

Please have a look at: https://github.com/kubernetes/kubernetes/pull/98952.

It does not implement this exactly (instead adding headers for the invoked kubectl command), but would show a way forward on implementation if the sig agrees.

[joshkurz]

Thanks @seans3 it looks like we could possibly leverage that code to allow for customHeaders. Do you think it would make sense to combine the functionality of ParseCommandHeaders and rename to ParseHeaders and just add a parameter that passes in the custom headers array. If they exist, we add those alongside of commandHeaders. High level idea, i'm sure there are some gotchas in there.

[liziwl]

Other k8s client, like fabric8 client or official Java client can add an interceptor (by okhttp) to set custom headers.

• How to add headers to OkHttp request interceptor?

[joshkurz]

Thanks @liziwl that is good to note.

So if other clients can do this, should kubectl be able too as well? Are there any concerns of doing this? What are the reasons why we would not want to do it?

[seans3]

/triage accepted

[brianpursley]

After talking about this in the monthly bug scrub today, we think something like this is better suited for a client-go credential plugin or custom client as opposed to implementing it in kubectl itself.

@joshkurz is there a specific need you are trying to satisfy with this request, or is it more of a general idea? If you can provide more information around the use case, it might help us better understand what you're trying to do.

/triage needs-information

[brianpursley]

/remove-triage accepted

[joshkurz]

@brianpursley TY for the triage and comments.

Overall the goal is to add a zero trust proxy in front of the k8s API that requires a custom header to accept traffic, which would then pass it along to the internal k8s api. Reason for this is because of zero day CVEs, like this https://www.cvedetails.com/cve/CVE-2018-1002105, which are avoided by not running the k8s api on the public internet.

I will continue to look for solutions outside of kubectl, possibly client-go auth plugin may allow me to set custom headers, but i'm still not sure after looking into the details of it. I believe it only sets an Authorization header and does not allow for customization of the headers being set.

[sftim]

/reopen

just to set metadata

[k8s-ci-robot]

@sftim: Reopened this issue.

In response to [this](https://github.com/kubernetes/kubectl/issues/1352#issuecomment-1430290961): >/reopen > >just to set metadata Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[sftim]

/close not-planned

[k8s-ci-robot]

@sftim: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/kubectl/issues/1352#issuecomment-1430291058): >/close not-planned Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[pjastrzabek-roche]

We have exactly the same use case. We don't want to implement custom authentication plugin, because the issue is not about changing authentication in any way. We do rely on various plugins already and the token generated is just fine.

As pointed out @joshkurz we also want to make it harder to reach out to certain clusters from the public network. Access is limited based on various premises, one of which is custom header (which in our case is a temporary access token that gives access to certain networks and has nothing to do with kubernetes authentication. We generate kubeconfig files on the fly and would be great if they could work with official kubectl. For that we need to set custom header only.

What I found is people forking kubectl just to have that option, which is easy to do, but in the long run problematic as we would need to own / distribute custom client to many users.

[mpuckett159]

/reopen /triage accepted Based on what I'm seeing after some Googling this is going to be an important feature for a lot of people. As noted in this issue there are several security solutions that use this type of workflow, and being able to enable this will help drastically reduce the impact of zero day CVEs by limiting the need to run API servers on the public internet.

Theoretically an unnecessarily complex solution could be to set up a transparent proxy server locally to inject headers for you, however, I think now that we have the addCmdHeaderHooks we have a usable pattern for implementing something more generic for users to add custom headers to their requests.

We don't have to necessarily handle it that way but I think we should implement something to handle this use case.

[k8s-ci-robot]

@mpuckett159: Reopened this issue.

In response to [this](https://github.com/kubernetes/kubectl/issues/1352#issuecomment-1681414680): >/reopen >/triage accepted >Based on what I'm seeing after some Googling this is going to be an important feature for a lot of people. As noted in this issue there are several security solutions that use this type of workflow, and being able to enable this will help drastically reduce the impact of zero day CVEs by limiting the need to run API servers on the public internet. > >Theoretically an unnecessarily complex solution could be to set up a transparent proxy server locally to inject headers for you, however, I think now that we have the [addCmdHeaderHooks](https://github.com/kubernetes/kubernetes/blob/9ee7185be6d613f2296bc10c6cb83406ee55fcee/staging/src/k8s.io/kubectl/pkg/cmd/cmd.go#L504) we have a usable pattern for implementing something more generic for users to add custom headers to their requests. > >We don't have to necessarily handle it that way but I think we should implement something to handle this use case. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[ecordell]

I didn't see this brought up anywhere:

https://kubernetes.io/docs/reference/access-authn-authz/authentication/#authenticating-proxy

Kubernetes has a built-in authn mode that determines the user and group from a request header. In general you likely won't be using kubectl to talk to a kube running in this authn mode in a production environment, but it's frustrating not being able to set the headers for dev and test clusters.