Create proxy connection (or other KubeAction)

[MortenMeisler]

Sorry for keep asking these "nub" questions :) But I can't figure out how to use the KubeActions (the enum values).

Trying to accomplish something similar to this for an existing pod: kubectl port-forward jupyter-notebook-7cfb95db6b-z2frd 8888:8888 -n default

Not sure if it's working though ( #10 )

Any example is much appreciated, thank you.

[tintoy]

I don’t think we’ve actually implemented port-forward to be honest; the pieces are there but nobody’s needed it so far so there is no explicit support for it. If you look at how the Exec method is implemented in KubeClient.WebSockets you could probably create a similar extension method for port-forwarding.

I might have time to look at implementing this sometime next week but the weekend is just about over in my time zone :)

[tintoy]

The enum is just a way for us to categorise the URLs we extracted from the K8s API Swagger document (which lists the API paths and their associated models).

[MortenMeisler]

Ah ok, thought I missed something. Makes sense, would be nice to have, think it could be useful to many who like me wants to "abstract" the kubectl to an internal webpage or similar.

I'll take a look at the exec method next week also.

Thanks for all the help, appreciate it :)

[MortenMeisler]

I've tried replicating the exec methods for websockets, but I get an 403 access denied when I try to use the portforward post command, not sure if the bearer token is passed on in the request header with this?

Here's the method (a bit hardcoded for now):

 public async static Task<WebSocket> PortForward(this IPodClientV1 podClient, string podName, string kubeNamespace = null, CancellationToken cancellation = default)
        {
            if (podClient == null)
                throw new ArgumentNullException(nameof(podClient));

            return await podClient.KubeClient.ConnectWebSocket(
                $"api/v1/namespaces/{kubeNamespace ?? podClient.KubeClient.DefaultNamespace}/pods/{podName}/portforward?ports=8888,8888",
                cancellation
            ).ConfigureAwait(false);
        }

//Initialize KubeApi client
            var Client = KubeApiClient.Create(new KubeClientOptions
            {
                ApiEndPoint = new Uri(myuri),
                AuthStrategy = KubeAuthStrategy.BearerToken,
                AccessToken = "mytoken",
                AllowInsecure = true // Don't validate server certificate
            });

            //throws: 403 forbidden
            var portforward = KubeOperations.PortForward(Client.PodsV1(), "jupyter-notebook-7cfb95db6b-788tm").GetAwaiter().GetResult();

I have no problems with other operations, like getting, listing, patching, creating.

[MortenMeisler]

Also, is there any good tools to sniff the traffic when I try this? It works fine with kubectl portforward and I can see the POST cmd when i run it verbose (--v=8), but I'd like to see the request when I run this in VS. Tried wireshark, but didn't really give me anything useful.

[tintoy]

What framework are you targeting with the client program?

On 27 Sep 2019, at 5:52 pm, Morten Meisler notifications@github.com wrote:

 I've tried replicating the websocket methods, but I get an 403 access denied when I try to use the portforward post command, not sure if the bearer token is passed on in the request header with this?

Here's the method (a bit hardcoded for now):

public async static Task PortForward(this IPodClientV1 podClient, string podName, string kubeNamespace = null, CancellationToken cancellation = default) { if (podClient == null) throw new ArgumentNullException(nameof(podClient));

        return await podClient.KubeClient.ConnectWebSocket(
            $"api/v1/namespaces/{kubeNamespace ?? podClient.KubeClient.DefaultNamespace}/pods/{podName}/portforward?ports=8888,8888",
            cancellation
        ).ConfigureAwait(false);
    }

//Initialize KubeApi client var Client = KubeApiClient.Create(new KubeClientOptions { ApiEndPoint = new Uri(myuri), AuthStrategy = KubeAuthStrategy.BearerToken, AccessToken = "mytoken", AllowInsecure = true // Don't validate server certificate });

        //throws: 403 forbidden
        var portforward = KubeOperations.PortForward(Client.PodsV1(), "jupyter-notebook-7cfb95db6b-788tm").GetAwaiter().GetResult();

I have no problems with other operations, like getting, listing, patching, creating.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[tintoy]

Does it work when you use Exec? Just trying to work out whether it’s a generic websockets issue or something specific to port-forward...

[MortenMeisler]

Does it work when you use Exec? Just trying to work out whether it’s a generic websockets issue or something specific to port-forward...

Yes I think so, this one went through and populated the multiplexer:

K8sMultiplexer multiplexer = Task.Run(async () => await Client.PodsV1().ExecAndConnect(
                        podName: "jupyter-notebook-7cfb95db6b-788tm",
                        kubeNamespace: "default",
                        command: "printenv",
                        stdin: true,
                        stdout: true,
                        stderr: true,
                        tty: true // Required for interactivity
                    )).GetAwaiter().GetResult();

            Console.WriteLine("Connected.");

I am targeting .NET Core 3.0 Preview 9. But seems like the other methods works fine from your library, so guessing I need something more/else for the POST call of portforward.

[MortenMeisler]

I should upgrade to release ofc, I just found out now it was released some days ago. Don't think it will matter here though.

[tintoy]

Not sure if it will help, but you could try enabling logging of payloads in the KubeClientOptions... that will at least show you what KubeClient thinks it’s sending...

[tintoy]

Hmm. Looks like port-forward uses something like POST /api/v1/namespaces/{namespace}/pods/{name}/portforward, but AFAIK WebSocket requests are supposed to start with a GET. I might need to check whether port-forward over WebSockets is actually supported by the K8s API...?

[MortenMeisler]

Ok thanks, I found this PR, seems like portforward is implemented over websockets: https://github.com/kubernetes/kubernetes/pull/33684 but maybe implementation can reveal something..

I tried logging, the websocket method is not passing in loggerfactory, so don't think it gives much (or maybe I need to set something more up).

Code used:

var loggerFactory = LoggerFactory.Create(builder =>
            {
                builder
                .SetMinimumLevel(LogLevel.Trace)
                    .AddConsole();
            });
            ILogger logger = loggerFactory.CreateLogger<ConsoleTester_App>();
            logger.LogInformation("Example log message");
            logger.LogDebug("Example debug message");

            //Initialize KubeApi client

            var Client = KubeApiClient.Create(new KubeClientOptions
            {
                ApiEndPoint = new Uri(baseUrl),
                AuthStrategy = KubeAuthStrategy.BearerToken,
                AccessToken = "mytoken",
                AllowInsecure = true, // Don't validate server certificate
                LogPayloads = true,
                LoggerFactory = loggerFactory
            }); ;

            K8sMultiplexer multiplexer = Task.Run(async () => await Client.PodsV1().ExecAndConnect(
                        podName: "jupyter-notebook-7cfb95db6b-788tm",
                        kubeNamespace: "default",
                        command: "printenv",
                        stdin: true,
                        stdout: true,
                        stderr: true,
                        tty: true
                        // Required for interactivity
                    )).GetAwaiter().GetResult();

            logger.LogInformation("Connected.");

            //403 forbidden
            var portforward = Task.Run(async () => await KubeOperations.PortForward(Client.PodsV1(), "jupyter-notebook-7cfb95db6b-788tm"));
            var portresult = portforward.GetAwaiter().GetResult();

Log:

info: ConsoleTester.ConsoleTester_App[0]
      Example log message
dbug: ConsoleTester.ConsoleTester_App[0]
      Example debug message
trce: KubeClient.Extensions.WebSockets.K8sMultiplexer[0]
      K8sMultiplexer created with 2 input streams (indexes: [1, 2]) and 1 output streams (indexes: [0]).
trce: KubeClient.Extensions.WebSockets.K8sMultiplexer[0]
      Message-send pump started.
trce: KubeClient.Extensions.WebSockets.K8sMultiplexer[0]
      Message-receive pump started.
info: ConsoleTester.ConsoleTester_App[0]
      Connected.
trce: KubeClient.Extensions.WebSockets.K8sMultiplexer[0]
      Received 1023 bytes for stream 1 (EndOfMessage = False).
trce: KubeClient.Extensions.WebSockets.K8sMultiplexer[0]
      Received 1024 additional bytes for stream 1 (EndOfMessage = False).
trce: KubeClient.Extensions.WebSockets.K8sMultiplexer[0]
      Received 457 additional bytes for stream 1 (EndOfMessage = True).
dbug: KubeClient.Extensions.WebSockets.K8sMultiplexer[0]
      Received first half of connection-close handshake (Status = NormalClosure, Description = ).
dbug: KubeClient.Extensions.WebSockets.K8sMultiplexer[0]
      Sending second half of connection-close handshake...
dbug: KubeClient.Extensions.WebSockets.K8sMultiplexer[0]
      Sent second half of connection-close handshake.
trce: KubeClient.Extensions.WebSockets.K8sMultiplexer[0]
      Message-receive pump terminated.
Unhandled exception. System.Net.WebSockets.WebSocketException (0x80004005): WebSocket connection failure.
 ---> System.Net.WebSockets.WebSocketException (0x80004005): Connection failure (status line = 'HTTP/1.1 403 Forbidden').
   at KubeClient.Extensions.WebSockets.K8sWebSocket.ParseAndValidateConnectResponseAsync(Stream stream, K8sWebSocketOptions options, String expectedSecWebSocketAccept, CancellationToken cancellationToken)
   at KubeClient.Extensions.WebSockets.K8sWebSocket.ConnectAsync(Uri uri, K8sWebSocketOptions options, CancellationToken cancellationToken)
   at KubeClient.Extensions.WebSockets.K8sWebSocket.ConnectAsync(Uri uri, K8sWebSocketOptions options, CancellationToken cancellationToken)
   at MyApp.Application.KubeOperations.PortForward(IPodClientV1 podClient, String podName, String kubeNamespace, CancellationToken cancellation) in 
--- End of stack trace from previous location where exception was thrown ---

[tintoy]

Hmm, ok, looks like I need to add some logging there :)

(sorry, don’t have a working cluster at the moment; I’ll see if I can get Docker for Desktop installed and have a go with that)

[MortenMeisler]

Np, thanks for the effort in any case :)

[tintoy]

Sure thing, sorry for the delay - I should be able to try this out tomorrow

[tintoy]

I’m still looking into this, will let you know what I find :)

[tintoy]

If you run kubectl with logging enabled and do an exec does it do a POST as well, or does it start off with a GET?

[tintoy]

And what value do you see in K8sWebSocketOptions.RequestedSubProtocols? Is it channel.k8s.io?

[tintoy]

Also, I take it if you don't do the exec call before the portforward call you still get the same result?

[tintoy]

You can also try KubeClientOptions.FromKubeConfig to ensure you're grabbing the same settings that kubectl uses.

[MortenMeisler]

Alright, I will try tonight when the kids are sleeping ;)