Service to debug cannot connect to required services within the cluster e.g. Redis / MySQL

[maxwagner-dev]

Problem: I have my app running on my machine (Windows 10, Docker 20.10.2, WSL Backend activated). Within my app (written in PHP) I want to connect to a redis instance to allow some caching. If I run that redis service locally everything works. But when I use Bridge for Kubernetes I want my app to connect to the running redis instance in my cluster. On my machine the redis service is available under "cluster-cache-redis-master" with port 6379, but the connection is always refused.

I can use the redis-cli to connect to a cluster. When I use port-forward to that redis instance and try to connect to it via redis-cli everything works. If I want to connect to the mapped Service via Bridge for Kubernetes the connection is refused.

The same error / failure exists for MySQL database.

Cluster Info:

• AKS with k8s version 1.17.7
• Istio with version 1.6

Thank you for the help in advance

[pragyamehta]

Hi @maxwagner-dev Bridge to Kubernetes does not work with Istio as of today, but we are looking into how to support that in the future. Could you elaborate on what you use istio for? That would help us prioritize its support. Meanwhile, could you try this scenario without istio enabled on your cluster?

[pragyamehta]

@maxwagner-dev On second thoughts, are you using B2K with isolation enabled or disabled? I believe this scenario should work for you with isolation disabled. Please let us know if that is not the case and provide us with logs.

[maxwagner-dev]

@pragyamehta We use istio for various reasons (Traffic controlling, traffic splitting for A/B Testing, Security...). Istio is essential to our product and we cannot live without it :) I saw some issues with Istio and Bridge for Kubernetes before here, that's why I have disabled isolation.

The PHP service is working and a get the debugger working, but only if I resolve the dependencies (MySQL Database, Redis) by my own (Running as different docker containers side by side with my PHP app).

If I go into my running docker container and ping the MySQL or Redis Service (Mapped via Bridge for Kubernetes), I got the right IP Address and everything, I can simple not connect to it.

[pragyamehta]

Hi @maxwagner-dev thanks for providing the information. There are a couple of things we can do to troubleshoot this issue.

1. Validate that we setup the connection successfully for you: For this, we would require the logs from your local machine. Attach logs from the following directory: For Windows: %TEMP%/Bridge to Kubernetes For OSX/Linux: $TMPDIR/Bridge to Kubernetes If you are a Visual Studio user, please also provide these logs: %temp%\Microsoft.VisualStudio.Kubernetes.Debugging If you are uncomfortable sharing them here, please send a mail attaching them to bridgetokubernetes@microsoft.com - please don't forget to add a link to this issue as well so that we have context about your issue :)

2. Tracking the request: Let's call the service being debugged as S and the pod under it as P. Once your locally running code makes the request to the redis/sql instance, the request should be forwarded to P (we replace the image running in P with our own image) - so we should check if there is a log in this pod when the request is made. To do that please run kubectl -n logs --follow=true This will keep the log window running and when you make the request, please observe if new logs come up and if you see any errors or not. It would be great if you could attach these in your email as well.

3. Comparing with our sample app which makes a similar connection: We have an existing getting started sample called Bikesharingweb application - Here is a link on how to use it. The link talks about how to debug the microservice Bikes. There is another microservice, called Users, which has a dependency on SQL running in the cluster itself and debugging Users should work. You can take a look at how it has been configured here and compare with how you have configured the connection in your code. Please try debugging Users (with your istio setup) after deploying the sample app using the first link I provided and see if that works for you. The readme file under Users would provide you with values with which you can configure the Bridge to Kubernetes connection. Let us know if this works for you.

Let me know if this makes sense. Looking forward to hearing from you with more details. Thanks!

[pragyamehta]

@maxwagner-dev another thing to try is below

4. Check connection to sql/exec from P: Exec into P and try to connect to sql/redis from inside P and let us know what you find.

[maxwagner-dev]

@pragyamehta Sorry for the late response. I setup some Test cases with the demo from the official docs

Test Case 1 (execute all steps from the Bikes AKS example - link here): If I run the complete example described in the tutorial, it works. So I have my full app deployment in the cluster and the node bikes app on my machine (Windows 10). I start the debugger in VSCode with the correct script and the node app will start running in my machine directly. Fine, that's how it should work :)

Test Case 2 (Run node bikes app in docker) Same steps executed like in Test Case 1, except I run the node app not directly on my machine, but instead in docker (WSL2 Backend / Hyper-V Backend - no difference). You can find the docker-compose setup at the end. Again start debugger in VSCode, but app cannot connect to database. Correct IP and Port is used, but connection is refused. I cannot see any logs in the pod in k8s.

Docker node bikes app log:

Docker-compose node bikes app:

services:
  bike_app:
    image: node:13.0.1
    container_name: bike-app-node
    entrypoint: "npm run-script debug"
    working_dir: "/usr/src/app"
    ports:
      - "3000:3000"
    volumes:
      - ../:/usr/src/app:rw

I have noticed that if I try to connect to a service like a MySQL database directly from windows e.g. with MySQL Workbench everything works and I can see the logs in the pod in my cluster. If I run any application inside a Docker container or try the same from within WSL the connection is always refused and no logs are written in the pod in my cluster. I can ping the service from Windows, Docker or WSL and get the correct IP, but simply cannot connect to it.

I have send you all log files from my tests to your linked email. I hope you can reproduce the Test Cases on your side and run into the same errors.

[lolodi]

Hi @maxwagner-dev I believe in this case the issue is not with Istio but with running your code containerized.

TL;DR: As of today Bridge does not support containerized workload, but we do have a work item tracking this and it's definitely on our radar.

Details: Bridge configures the network on the local machine to "bridge" your machine and the cluster. This involves creating port fowardings to map the remote services to local IPs on your dev box, and adding entries to the Host file so that dns resolution points to these IPs. When you contact the MySQL DB natively from your Windows machine, your call goes through the port forwarding, makes it to the cluster and then gets redirected to the DB running there. If you do that from within the container the call gets lost because the Host file in the container doesn't contains the entries, so you don't get resolved to the local IPs, but also because the container network is not the same as your machine, so even if you try to call the local IP for the MySQL DB it wouldn't work.

Hope this adds a little more clarity :)

[maxwagner-dev]

Ok, thank you. Looking forward to this feature.