Seems to not successfully authenticate against local kube with default config

[justinmchase]

KubeConfigRestClient.forKubeConfig(await KubeConfig.getDefaultConfig());

When I log what its loading it its correctly printing this object (all keys are correct):

PatchedKubeConfigRestClient {
  ctx: KubeConfigContext {
    context: { cluster: "docker-desktop", user: "docker-desktop" },
    cluster: {
      "certificate-authority-data": "<redacted>",
      server: "https://kubernetes.docker.internal:6443"
    },
    user: {
      "client-certificate-data": "<redacted>",
      "client-key-data": "<redacted>"
    }
  },
  httpClient: HttpClient { rid: 4 },
  defaultNamespace: "default"
}

But when I attempt to make a request it gets this error:

Error: Kubernetes says: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "example"

If I just use await autoDetectClient(); it falls back to the KubectlRawRestClient and then gives me 403 on certain apis such as exec.

[justinmchase]

It doesn't seem to ever be using the user client certificate / key. It is connecting to the server, it just thinks I'm anonymous instead of the user in the context.

[danopia]

I don't have a great tracking ticket for this, but I believe the issue you're seeing is a lack of client certificate support in the fetch API. (In a browser, client certificate are a UI matter; Deno doesn't have that UI context)

If Deno has gained client certificate support, it definitely should be wired up here.

This is one of the reasons that KubectlRawClient is leaned upon so much. Deno's http client abilities are a bit lacking compared to golang for example. I thought I added an explanatory log line when unsupported features are used so I'll have to check in on that.

As a general workaround, run kubectl proxy and then this library should have no problem talking to that localhost endpoint. And if you're running in a cluster then the service account token should work fine.

Thanks for reporting; there's something to do here for sure even if it's just documentation work :)

[justinmchase]

That tracks with what I discovered also.

Here is the best ticket I could find: https://github.com/denoland/deno/pull/9426

I think its not yet implemented but it looks like there is an existing pr which is "pretty close".

[justinmchase]

Also, it seems like the proxy doesn't support websocket apis directly... This is admittedly an area I am not an expert in an have just taken libraries for granted but after hacking around in here a lot it seems like it just doesn't work to go through the proxy for ws based apis (pod/attach is also a ws api).

When I load up the kubernetes dashboard, which runs through the proxy, and I go to a pod and click the exec into pod button

It then makes a curious call into a "sockjs" api which appears to use standard headers to get a token...

And then it calls into the websocket api through that same "sockjs" route but it attaches the token it got from the other route to the request url.

I haven't gone down the rabbit hole of trying to reproduce this but it might be the only way to make it work via proxy, I'm not sure.

Here's clues: https://github.com/kubernetes/dashboard/search?q=sockjs

[danopia]

I'm seeing if I can revive this issue into a TLS client cert tracking issue: https://github.com/denoland/deno/issues/10516
I'll note that the client certificate issue you linked is for connectTls() specifically, not fetch(), so if that did merge, one would have to handle the inner HTTP/WS protocol within TypeScript :)

The other blocker I am tracking is Deno's inability to connect to IP addresses over TLS, e.g. https://1.1.1.1 fails. GKE clusters need this for external access. For these reasons I really try to keep the KubectlRawRestClient as functional as possible; I need it for my workstation!

Maybe it would make sense for this library to include a curl based RestClient. Like using Deno.run(). It could even do websocket apparently, with some micromanagement: https://gist.github.com/htp/fbce19069187ec1cc486b594104f01d0

---

Regarding kubectl proxy, I think what you're seeing with sockjs is a red herring; in fact your screenshot shows that the proxy can pass through websockets just fine. However because the requests are to the /proxy/ route of kubernetes-dashboard, the URLs are whatever the kubernetes-dashboard service exposes, and as you say kubernetes-dashboard just connects to its own SockJS endpoint. This is an implementation detail of that project and is likely related to the browser usecase. SockJS allows emulation so if the user's firewall doesn't allow WebSocket, they can still exec into their pods.

I'm not sure if you already have some code uploaded to try doing one of these websocket pod/exec calls.. If you do then I could try running/fixing that sooner than later. Overall I can't really promise when I'll be able to properly tackle this overall usecase :(

[danopia]

I see that I have a fail-fast for the Deno fetch limitation I personally hit with GKE: https://github.com/cloudydeno/deno-kubernetes_client/blob/3864e127de51ff4a246cfa4c09412d19164797f9/transports/via-kubeconfig.ts#L67-L72

I suppose the most relevant actionable for this issue as filed is adding another fail-fast if the kubeconfig includes a client certificate. ~Just need a proper Deno tracking issue to link to :)~ #10516 is reopened now

[justinmchase]

Yeah I agree, there is basically nothing you can do in that case until deno supports client certificates.

[justinmchase]

Related To https://github.com/denoland/deno/issues/10516

[justinmchase]

@danopia I'm not sure if you already have some code uploaded to try doing one of these websocket pod/exec calls...

I had something but got hit by road blocks at every angle and couldn't get it to work. There were so many issues it was hard to isolate why it wasn't working but I was constantly getting 401 errors from the api when trying to run the exec command with a websocket and if you do a plain GET/POST without trying to initiate a websocket it just returns a success status code but doesn't do anything.

The three big blockers here are:

1. Deno doesn't support client certificates yet
2. Denos websocket implementation doesn't allow attaching custom headers, including the Authorization header, which is what kubernetes uses to authenticate websocket connections.
3. Deno doesn't allow TLS with ip addresses
4. The pod/exec api returns a 401 through the proxy or direct method

I got some traction from the Deno team regarding 2 here. I used the line of reasoning that the standard they are attempting to adhere too is noble but ultimately its desinged for a browser and since Deno is a primarily backend system it may be reasonable to deviate from the standards slightly when appropriate.

Here is a related conversation with the standards group, trying to convince them to support headers (or at least the one header) but they appear to be staunchly against it.

Here is a workaround to 2: https://github.com/Scientific-Guy/custom-socket

Which I think works but its hard to test because my local kube needs client certificate support and my remote cluster needs IP support so its just a pain.

Conclusion

You could add the websocket workaround, that @Scientific-Guy published, then update the pod exec api to use the websocket with an auth header. That would work when running InCluster (or anytime you have a user token).

Also, the fail fast when the user has a client certificate would help.

Otherwise we're stuck waiting for client certs, tls+ip support and hopefully an update to the websocket api to support headers.

[danopia]

Please use whatwg/html#3 for discussion specifically about WebSockets (aka the exec or attach APIs)

[danopia]

Deno just merged cert auth for fetch() so I opened whatwg/html#7 to track implementing it.

[danopia]

Fun little discovery, Kubernetes supports passing the auth token on websockets via an abuse of the subprotocols system. So that's an unblock for in-cluster websockets without Deno needing to do non-spec things.

[justinmchase]

Do you mean thats a "blocker" I'm not sure I understand what you mean by "unblock"?

But yeah that is the cruxt of this conversation, previously linked: whatwg/websockets#16

Also, I'd argue that its not an abuse of the subprotocols but actually part of the spec and the standard is what is varying from the spec. The nodejs websocket does support it... But Deno is taking a hard(er)-lined stance than node about sticking with official html and standards.

[danopia]

This is unrelated to all the header stuff. Kubernetes specifically has a proprietary way to embed a bearer token into the subprotocols field. So you don't actually need to attach arbitrary headers to authenticate WebSockets to Kubernetes.

[justinmchase]

Interesting, so that's why you're saying it could unblock this? So basically you establish the connection without the auth header and then you embed the token in the subsequent handshake messages? That sounds very promising.

[danopia]

Unblocks websockets w/ bearer auth, yea. I haven't fully tested it yet but here's my diff that adds the support: https://github.com/cloudydeno/deno-kubernetes_client/pull/6/commits/6d5f208edd7aa71ad092c13222f87b8aa27812d8

The gist is that a base64url.bearer.authorization.k8s.io.${base64urlencode(bearerToken)} subprotocol will be understood by Kubernetes. I dislike the approach but it's been implemented for years now