`pod-exec` Fails with 403

[drcariel]

When using the following command: api.v1.namespaces('kube-system').pods(albPod.name).exec.get({ qs: { command: ['nginx', '-t'], container: 'nginx-ingress', stdout: true, stderr: true } }); I get the following error:

{ Error: Unexpected server response: 403
    at ClientRequest.req.on (node_modules/ws/lib/websocket.js:535:5)
    at emitOne (events.js:116:13)
    at ClientRequest.emit (events.js:211:7)
    at HTTPParser.parserOnIncomingClient (_http_client.js:544:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:117:17)
    at TLSSocket.socketOnData (_http_client.js:440:20)
    at emitOne (events.js:116:13)
    at TLSSocket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:250:11) messages: [] }

This is the same failure for both get and post. I have tested using config.fromKubeconfig(process.env.KUBECONFIG) but have yet to try using config.getInCluster() but I do not think this is an auth issue. I am able to successfully use kubectl to get the nginx -t output like so: kubectl exec -in kube-system albPod.name -c nginx-ingress -- bash -c 'nginx -t'

My only guess is something to do with the websocket upgrade. I noticed the upgrade response that kubectl uses is the following:

Response Headers:
Connection: Upgrade
Upgrade: SPDY/3.1
X-Stream-Protocol-Version: v4.channel.k8s.io

while the code included here uses for pod-exec is using base64.channel.k8s.io.

[silasbw]

The exec code is definitely experimental. What version of Kubernetes are you running?

[lancehall123]

I have the same problem. Here is my version. I noticed when debugging that "parsedUrl.auth" is undefined. Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.9-gke.5", GitCommit:"d776b4deeb3655fa4b8f4e8e7e4651d00c5f4a98", GitTreeState:"clean", BuildDate:"2018-11-08T20:33:00Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

[gempesaw]

Hmm, I didn't really think about auth at all when I was setting that up, so it's very possible I missed something. Perhaps I can find some time to look into it...

[lbogdan]

I also got bitten by this.

After a fair bit of debugging, I realized that while client certificate authentication worked fine, token authentication failed. This is because "normal" http(s) requests are done using request, which uses options.auth(.bearer) for token authentication, while the ws module uses http(s) core modules, and only supports setting an Authorization: Bearer sometokenhere header for token authentication, which kubernetes-client doesn't handle.

My quick&hacky workaround for now is to add

  options.headers = {
    ...options.headers,
    authorization: `Bearer ${options.auth.bearer}`
  }

just before https://github.com/godaddy/kubernetes-client/blob/847623ccfe5806924976d815985fa9de724a4338/lib/backends/request.js#L62

[feiniao0308]

When this issue can be fixed?

[silasbw]

I would love to take a PR for this :)

We're busy working on merging this client with the kubernetes-client/javascript one, so I apologize for being unresponsive on this issue.

[feiniao0308]

Thanks @silasbw . Looking forward to seeing the fix in the coming release. :-)

[Stono]

Any chance we can get the fix merged?

[kassner]

I managed to work around this issue with some user code (as if you can't manually edit the library's code).

        const res = await client.api.v1.namespaces(namespace).pods(podName).exec.post({
            qs: {
                command: ['ls', '-la', '/']
            },
            headers: {
                authorization: `Bearer ${client.backend.requestOptions.auth.bearer}`
            }
        })

I'm using Kubernetes 1.13.