Cannot Access AWS RDS from App Mesh

[nitinkapur]

We have a AWS EKS Fargate cluster with a service that connects to RDS DB and ElasticCache on AWS. I have setup a mesh with Egress traffic allowed and deployed a virtual node and virtual service.

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualService
metadata:
  name: ganesh-permissions
  namespace: dev
spec:
  awsName: ganesh-permissions.dev.svc.cluster.local
  provider:
    virtualNode:
      virtualNodeRef:
        name: ganesh-permissions-vnode
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: ganesh-permissions-vnode
  namespace: dev
spec:
  podSelector:
    matchLabels:
      app: ganesh-permissions
  listeners:
    - portMapping:
        port: 80
        protocol: http
  backends:
    - virtualService:
        virtualServiceRef:
          name: ganesh-permissions
  serviceDiscovery:
    dns:
      hostname: ganesh-permissions.dev.svc.cluster.local

I can now access the service api through load balancer and there is a version number call that I can make which returns the version number and a 200 trace is available in X-Ray. After this there is a call that I make to get some data from the service which uses an authentication header to authenticate the user and then should pull the data and return results. But this call fails and I do see a 504 error logged in the X-Ray traces.

Now either the service is not accepting headers or it is having issue with connecting to the database. How to make sure it can connect to the DB? The pod uses a connection string to connect to the AWS RDS MS SQL Server

[karanvasnani]

Hi @nitinkapur, does the connection to RDS MS SQL has TLS enabled or not? Additionally, is it possible for you to provide logs from Envoy as that would help us narrow down the problem.

I suspect connection to MS SQL Server might also be impacted by this bug (https://github.com/aws/aws-app-mesh-roadmap/issues/62) identified originally for MySql connectivity. Is it possible for you to try the workaround from that issue or as mentioned below (I believe the ignored port here would be 1433). We're in the process of releasing a fix for that bug. Meanwhile, I'll try to reproduce this on my end.

Workaround

To work around this issue, add port 1433 to the list of ignored egress ports for Envoy.

For ECS, you can set this in the Proxy Configuration.

For EKS, you can set the environment variable APPMESH_EGRESS_IGNORED_PORTS in the pod definition for the proxyinit container. See Getting Started with EKS for more information on this definition.

For EC2, you can set APPMESH_EGRESS_IGNORED_PORTS using the proxy route manager script. See https://docs.aws.amazon.com/app-mesh/latest/userguide/mesh-getting-started-ec2.html for more information on this script.

[nitinkapur]

What resolved it was to allow Egress traffic from the Mesh.

[cid:5118a800-4f83-45ef-af0f-58d24f3300f6]

---

From: Karan Vasnani notifications@github.com Sent: Thursday, October 8, 2020 6:58 PM To: aws/aws-app-mesh-roadmap aws-app-mesh-roadmap@noreply.github.com Cc: nitinkapur nitinkapur@live.com; Mention mention@noreply.github.com Subject: Re: [aws/aws-app-mesh-roadmap] Cannot Access AWS RDS from App Mesh (#270)

Hi @nitinkapurhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnitinkapur&data=02%7C01%7C%7C69b09c96b5b74a19cdf908d86bddb3c8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637377947225997917&sdata=YY27jSLjVrMb6RnFR1u0st5Js7QqhFcIajCmJUFtBOw%3D&reserved=0, does the connection to RDS MS SQL has TLS enabled or not? Additionally, is it possible for you to provide logs from Envoy as that would help us narrow down the problem.

I suspect connection to MS SQL Server might also be impacted by this bug (#62https://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faws%2Faws-app-mesh-roadmap%2Fissues%2F62&data=02%7C01%7C%7C69b09c96b5b74a19cdf908d86bddb3c8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637377947225997917&sdata=27aDzhIvXBvQITwDZnOouZnFB5PuJcJVR3F4%2FEpK0A4%3D&reserved=0) identified originally for MySql connectivity. Is it possible for you to try the workaround from that issue or as mentioned below (I believe the ignored port here would be 1433). We're in the process of releasing a fix for that bug. Meanwhile, I'll try to reproduce this on my end.

Workaround

To work around this issue, add port 1433 to the list of ignored egress ports for Envoy.

For ECS, you can set this in the Proxy Configurationhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2FAmazonECS%2Flatest%2FAPIReference%2FAPI_ProxyConfiguration.html&data=02%7C01%7C%7C69b09c96b5b74a19cdf908d86bddb3c8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637377947226007912&sdata=21ILvuh3pXXxnkURHs82KbrZPflDILc28fiiGVHf72w%3D&reserved=0.

For EKS, you can set the environment variable APPMESH_EGRESS_IGNORED_PORTS in the pod definition for the proxyinit container. See Getting Started with EKShttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Feks%2Flatest%2Fuserguide%2Fmesh-gs-k8s.html&data=02%7C01%7C%7C69b09c96b5b74a19cdf908d86bddb3c8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637377947226007912&sdata=qAbNsT2%2BZ4xpe8RXIVm7lD6qYZ8vBRkLcEviH43mWxE%3D&reserved=0 for more information on this definition.

For EC2, you can set APPMESH_EGRESS_IGNORED_PORTS using the proxy route manager script. See https://docs.aws.amazon.com/app-mesh/latest/userguide/mesh-getting-started-ec2.htmlhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.aws.amazon.com%2Fapp-mesh%2Flatest%2Fuserguide%2Fmesh-getting-started-ec2.html&data=02%7C01%7C%7C69b09c96b5b74a19cdf908d86bddb3c8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637377947226007912&sdata=aZmayOu4sPDgqV9VRScGC703fmBm3qQKGKjLP4wEVis%3D&reserved=0 for more information on this script.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faws%2Faws-app-mesh-roadmap%2Fissues%2F270%23issuecomment-705865649&data=02%7C01%7C%7C69b09c96b5b74a19cdf908d86bddb3c8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637377947226017906&sdata=7kJBdA9IME3lqFWVPbhsJK%2FGVn6XSKygf5WCs7M6BCU%3D&reserved=0, or unsubscribehttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACSHNZY52CKQ74XFN5FESYLSJY72DANCNFSM4SFRBL5Q&data=02%7C01%7C%7C69b09c96b5b74a19cdf908d86bddb3c8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637377947226017906&sdata=69TTfKo70QWRh%2BxkZOOG%2FG44iVNrt63NYGtNXOfVT6I%3D&reserved=0.

[karanvasnani]

Thanks for the update. Closing this issue then.

[suniltheta]

The workaround on our troubleshooting docs is updated. Please refer the official AWS docs:

https://docs.aws.amazon.com/app-mesh/latest/userguide/troubleshooting-connectivity.html#ts-connectivity-troubleshooting-mysql-and-smtp

What was missing earlier was even though we use App Mesh image Envoy version 1.15.0 or later, we are still not able to connected to the RDS instances from the application node. The caveat here is that you should not model external services such as MySQL, SMTP, MSSQL, etc. as a backend for your application's virtual node.

[mostafafarzaneh]

Thanks, @suniltheta. I have a service in Appmesh that wants to connect to RDS endpoint. Appmesh EgressFilter has been set to DROP_ALL. According to this, Appmesh allows any traffic to *.amazonaws.com. But I cannot connect to RDS service. All I know is the golang client says bad connection. The RDS endpoint is ended with rds.amazonaws.com

Surly ignoring the RDS port(3306) will resolve the problem. I suppose Appmesh should allow connections to *.amazonaws.com. Is that right?

I am using aws-appmesh-envoy version v1.22.0.0-prod and I have not modeled the RDS as a backend to our virtual node.

[suniltheta]

Sorry missed this under the radar. Connection to *.amazonaws.com is only possible over port 443.