Support raw TCP proxying

[lscpike]

Is your feature request related to a problem? Please describe. I have a container that hosts a custom tcp protocol in addition to a http server. I would like to be able to debug this service locally using B2K and have services in the cluster connect to my custom tcp endpoint.

Describe the solution you'd like The B2K proxy deployed to the cluster in place of my pod should proxy all the tcp ports from the container to the same ports on my local process.

Describe alternatives you've considered N/A

Additional context I'm not sure if this should already work. I have been unsuccessful in my attempts. It appears the proxy is http only and proxies directly to only the ASPNETCORE_URLS endpoints. Is that correct?

[daniv-msft]

(+ @amsoedal who is looking into customer issues this week to follow up)

Thanks @lscpike for opening this issue. Our proxy is working at the TCP level, so from what you describe we should work.

At first sight, it seems that there are two potential issues here:

• Possibility 1 - We don't detect properly the TCP ports from the container, and so we never proxy them.
• Possibility 2 - We do detect properly the ports, but for some reasons our proxy doesn't intercept the traffic properly.

To validate what happens, could you please:

• Connect to your cluster using Bridge.

• Run kubectl logs <POD_NAME> for the pod where our remote agent is running (Docker image: bridgetokubernetes.azurecr.io/lpkremoteagent:0.1.7).

• In the logs, you should see a line like ReversePortForwardConnector start listening on port 80.

• Can you see your TCP ports listed there?

Feel free to attach the logs to this issue/send them to us by email (BridgeToKubernetes@microsoft.com) if you prefer us to have a look at them directly.

[lscpike]

It's great to hear the proxy is working at the TCP level. This may just be something I've not configured correctly then.

The test set up is:

• Docker Desktop (Windows) with built in K8s
• Bridge to Kubernetes version: 2.1.20210515.1
• Default ASPNET Core Web API template (WeatherForecast).
• Added a TCP server as a HostedService that simply echos lines. This is tested via telnet.

using System.IO;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

namespace TestBridgeToK
{
    public class TcpServer : BackgroundService
    {
        private const int Port = 8003;
        private readonly ILogger<TcpServer> _logger;

        public TcpServer(ILogger<TcpServer> logger)
        {
            _logger = logger;
        }

        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            var tcpListener = new TcpListener(new IPEndPoint(IPAddress.Any, Port));

            try
            {
                _logger.LogInformation("TCP server starting on {port}", Port);
                tcpListener.Start();

                while (!stoppingToken.IsCancellationRequested)
                {
                    var client = await tcpListener.AcceptTcpClientAsync();

                    using var stream = client.GetStream();
                    _logger.LogInformation("Accepted client connection from {RemoteEndpoint}", stream.Socket.RemoteEndPoint);

                    using var streamWriter = new StreamWriter(stream);
                    using var streamReader = new StreamReader(stream);

                    string line;
                    while ((line = await streamReader.ReadLineAsync()) != null && !stoppingToken.IsCancellationRequested)
                    {
                        await streamWriter.WriteLineAsync("Echo: " + line);
                        await streamWriter.FlushAsync();
                    }

                    _logger.LogInformation("Client disconnected from {RemoteEndpoint}", stream.Socket.RemoteEndPoint);

                }

            }
            finally
            {
                _logger.LogInformation("Stopping TCP server");

                tcpListener.Stop();
            }

        }

    }

}

Here's what I have tried so far:

1. Having both HTTP (80) and TCP (8003) ports defined in at both the K8s container and the service (ClusterIP). This appears to proxy only the first port it sees on the container and always to the HTTP port.

2. Having only the TCP port on the K8s container and service. This still proxies the TCP port to the HTTP port. This suggests B2K is looking explicitly for the http port of the process.

3. Adding explicit Kestrel config with a URL at port 80. This causes the Kestrel server to ignore the ASPNETCORE_urls and binds to 80 correctly. The proxy now seems to not be aware of which port the HTTP server is on and fails to proxy any connections. I'm assuming it is looking at the env variable only.

4. Changing the project to a normal SDK <Project Sdk="Microsoft.NET.Sdk">. This stops the ASPNETCORE_urls env variable being added (I'm guessing this is some feature of the VS debugger.) The proxy now does not listen on any port and the log is empty.

In conclusion it appears B2K is only looking for the ASPNETCORE_urls and proxying the first port on the container to it. Does that sound correct? If so how do I get it to proxy other tcp ports?

Thanks very much for your help.

[lscpike]

I've had some success. It looks like if I set the ASPNETCORE_URLS env variable to pretend that the custom tcp port is a http url then B2K will proxy it. This requires me to have the actual kestrel urls set in configuration so that ASPNETCORE_URLS is ignored (else kestrel will bind the port I need for the custom tcp server).

One problem is B2K seems to map the container ports to the ASPNETCORE_URLS ports in the same order order. Therefore, the order must be the same in both else the ports get cross wired.

It feels like B2K should just be proxying the container ports 1-1 to the local service. It seems like a sensible expectation that the local process is using the same ports as the deployed container.

Is there some other way to tell B2K which ports to proxy?

[daniv-msft]

Thanks @lscpike for the extra information. We cannot map 1-1 the ports from the container to the local service, because in a lot of cases the ports used there are different (typically, port 80 is often used inside the cluster while any default port is used locally).

From what you describe, I believe you're using Visual Studio, right? On Visual Studio, our extension automatically detects the port to use and I'm not aware of a way to override that. @danegsta Would you know if there is a way to set the port to use? I looked into the .csproj.user file, but the port isn't stored there.

If using VS Code is a possibility for you, we let customers set their own port there during the configuration. Would this be a possibility?

[lscpike]

Yes I've been using Visual Studio. I'll give vs code a try but ultimately its a down grade from the full visual studio.

I think there is definitely a feature request here for the vs extension to define the port mappings in some way. No need for a UI. Maybe use the KubernetesLocalProcessConfig.yml? Shall I open a new ticket to request that?

On a separate note, I don't think B2K is an option for me after all. I have an Orleans project that I wanted to use this on. If you are not familiar, Orleans is a distributed actor framework which has the concept of a cluster of nodes. The nodes (silos in Orleans language) communicate directly via tcp to endpoints shared via a membership protocol. These endpoints are the pod IPs and obviously don't provide a way for B2K to proxy them back into the k8s cluster. I think I know the answer, but are there any ways B2K could work here?

[amsoedal]

Hi @lscpike, thanks for your question about Orleans. I'm not familiar with this framework, but if you need to make the pod IPs accessible to the local process and proxy traffic to them via the cluster, we do support adding generic endpoints to the KuberentesLocalProcessConfig.yaml file with the "externalendpoints" keyword. For example:

env:
  - name: DB_HOST
    value: $(externalendpoints:server-bridgetest123.database.windows.net:1433)

Would something like this work for your scenario? I'll need to double check that we support IPs as well as dns names for this. Also regarding your other question:

I think there is definitely a feature request here for the vs extension to define the port mappings in some way. No need for a UI. Maybe use the KubernetesLocalProcessConfig.yml? Shall I open a new ticket to request that?

I've logged a work item on our side so no need to open a new ticket. Thanks for the suggestion!

[lscpike]

I get an error 'loading KuberentesLocalProcessConfig.yaml' saying unknown token type 'externalendpoints' with the above. What is externalendpoints supposed to do?

if you need to make the pod IPs accessible to the local process and proxy traffic to them via the cluster

That pretty much sums up what I need. Orleans shares "gossip" about cluster members which provides the pod ip of the member. The advertised ip is overridable which may offer some work around, but I can't see how.

[lscpike]

I tried in VS Code and the externalendpoints works. I don't think it will help in this situation though.

Orleans clustering is just incompatible with B2K so we'll try switching to a non clustered deployment for local dev.

Thank you very much for the help. I'll close this ticket, but feel free to reopen if you need it to drive the feature request.

[amsoedal]

Alright, thanks for giving us a try! Appreciate it.