403 Error when opening remote shell into container

[gheinrich]

Hello, thank you very much for this very useful tool! When I try opening a remote shell into any of my pods I get a 403 error back. I am wondering if you know why that might be?

The error I see in kubebox is:

Error: Failed to get resource /api/v1/namespaces/<xxx>/pods/<xxx>/exec?stdout=1&stdin=1&stderr=1&tty=1&container=main&command=%2Fbin%2Fsh&command=-c&comma │
│nd=TERM%3Dxterm+%24%28+%28type+getent+%3E+%2Fdev%2Fnull+2%3E%261+%26%26+getent+passwd+root+%7C+cut+-d%3A+-f7+2%3E%2Fdev%2Fnull%29+%7C%7C+echo+%2Fbin%2Fsh%29, status code: 403

On the other hand I am able to open a shell into the main container using kubectl exec -it <xx> /bin/sh.

Thanks!

[gheinrich]

I noticed that kubectl uses a POST and kubebox uses a GET for the exec endpoint.

This fixed it for me:

@@ -176,7 +176,7 @@ class Client {
     command.forEach(c => path.addQuery('command', c));
     return merge({
       path    : path.toString(),
-      method  : 'GET',
+      method  : 'POST',
       headers : {
         // https://tools.ietf.org/html/rfc6455
         Connection               : 'Upgrade',

I can send a Pull Request.

[astefanutti]

Thanks a lot for the feedback and report.

That's a good catch. Sure, you can send a pull request. I'll merge it right away.

[astefanutti]

I've just tested the change and I face some 405 errors on some setups. I'm still trying to understand what could cause 405.

kubectl uses POST, but it uses the HTTP/2 streaming protocol, as opposed to Kubebox which uses the WebSocket protocol (both for terminal and Web clients).

Out of curiosity, what version / setup of Kubernetes do you use?

[astefanutti]

The 405 error is returned by https://github.com/golang/net/blob/adae6a3d119ae4890b46832a2e88a95adc62b8e7/websocket/hybi.go#L492-L494, while the server is upgrading the connection to Web socket.

That still raises the question why you gets the 403 error for the GET method on the exec endpoint (could be just an RBAC thing), and how it can possibly be working for the POST method.

Let me re-open that issue so that we can get the bottom line of this.

[gheinrich]

Sorry to hear the patch is causing troubles! I am using kubectl 1.12 and my cluster has API 1.10.

[astefanutti]

No worries. Thanks for the info.

I confirm using POST is working with Kubernetes version 1.10.10 but fails on 1.12. I need to investigate further and keep you posted.

[astefanutti]

After a deeper look at it, the following change kubernetes/kubernetes@174b6d0e2fc99d9964a7d5a7484aa0b7d50b4be1, introduced in version 1.11, is responsible for the difference.

Before, the request was always redirected and a new GET request was actually being issued by the API server to access the container runtime directly.

After, the request between the API server and the container runtime is proxied by the kubelet, so the POST method gets proxied and the WebSocket handshake fails at https://github.com/golang/net/blob/adae6a3d119ae4890b46832a2e88a95adc62b8e7/websocket/hybi.go#L492-L494.

It means that, starting k8s version 1.11, WebSocket based POST requests to the exec endpoint are broken.

So that leaves two options:

• Either migrate from using WebSocket to using HTTP/2, though I'm not sure this is possible for the Web version;
• Or understand why the GET method returns 403 while the POST is not, possibly an issue with RBAC, as it works on my side with k8s version 1.10.10.

[astefanutti]

I had another look at it and it turns out kubectl uses SPDY (and not HTTP/2), which is deprecated and whose support is planned to be dropped (kubernetes/enhancements#384). Until support for HTTP/2 is delivered (kubernetes/kubernetes/issues/7452), it seems moving kubectl to WebSocket is favored (kubernetes/kubernetes#48633).

Based on that, I'd be inclined to revert the change to use POST and keep that issue open to:

• Understand why 403 is returned in your setup for GET requests and not for POST ones,
• Fix broken WebSocket based POST requests to the exec endpoint, starting k8s 1.11, because of kubernetes/kubernetes@174b6d0.

[astefanutti]

@gheinrich by chance, would you be able to test on a newer version of Kubernetes, like 1.11 or 1.12? It may be that the issue doesn't exist in newer versions.

[astefanutti]

Let me speculatively close this. Feel free to re-open if you still face the issue with the latest release version.