search

aws / apprunner-roadmap

This is the public roadmap for AWS App Runner.
https://aws.amazon.com/apprunner/
Other
292 stars 13 forks source link

AppRunner-RDS connection issues #167

Closed MrKappa closed 1 year ago

MrKappa commented 1 year ago

I'm deploying an AppRunner service using an ECR Image, this service is public (both outgoing and incoming) and the actual issue is that I can't connect to a public RDS database. Actually RDS database is public just for debugging purposes and rapid testing of the image but my application can't reach (ETIMEOUT) that public database. Database endpoint is public and sg is allowing all inbound and all outbound. The same image deployed in ECS Fargate works correctly and also in my local environment while pointing to the public RDS instance.

Is this an issue or am I missing something?

amitgupta85 commented 1 year ago

Are you using a VPC Connector with the App Runner service? If yes, please make sure that private subnets (which route traffic outside the VPC via NAT Gateway) are configured for the VPC Connector. It does not work with public subnets which route traffic via internet gateway - https://docs.aws.amazon.com/apprunner/latest/dg/network-vpc.html If this doesn't solve the problem, we would like to know the service ARN to help debug this further.

MrKappa commented 1 year ago

Hi,

no, I'm not using a VPC connector because the RDS instance is public (temporary, just for testing).

AppRunner Network config: image

RDS: Publicly accessible: Yes

AppRunner ARN: arn:aws:apprunner:eu-west-1:257216940263:service/test-portal/8e6dd18f2c0b4d8aa7bf9dcbfce20aa1 RDS Resource ID: db-IRMCBNNDIWUR5Y4AKHUYXJFWUI

Thank you and let me know if you need further info.

helihang commented 1 year ago

Hi MrKappa,

I took a deep into logs on service side. The application task launched but failed at health check. Do you mind to share more application logs and RDS error logs when the RDS fails on connection? Please remove any sensitive data in the logs.

Meanwhile, App Runner service is creating ENIs for service to access resources under custom VPC when using VPC connector (https://aws.amazon.com/blogs/containers/deep-dive-on-aws-app-runner-vpc-networking/). Can you verify if the same connection issue exist while using App Runner service with VPC connector? Thanks.

MrKappa commented 1 year ago

Hi helihang,

here the application logs:

01-17-2023 09:50:20 AM [Nest] 1 - 01/17/2023, 8:50:20 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (6)... 01-17-2023 09:50:07 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:50:07 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:50:07 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:50:07 AM Error: connect ETIMEDOUT 01-17-2023 09:50:07 AM [Nest] 1 - 01/17/2023, 8:50:07 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (5)... 01-17-2023 09:49:54 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:49:54 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:49:54 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:49:54 AM Error: connect ETIMEDOUT 01-17-2023 09:49:54 AM [Nest] 1 - 01/17/2023, 8:49:54 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (4)... 01-17-2023 09:49:41 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:49:41 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:49:41 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:49:41 AM Error: connect ETIMEDOUT 01-17-2023 09:49:41 AM [Nest] 1 - 01/17/2023, 8:49:41 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (3)... 01-17-2023 09:49:28 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:49:28 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:49:28 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:49:28 AM Error: connect ETIMEDOUT 01-17-2023 09:49:28 AM [Nest] 1 - 01/17/2023, 8:49:28 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (2)... 01-17-2023 09:49:15 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:49:15 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:49:15 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:49:15 AM Error: connect ETIMEDOUT 01-17-2023 09:49:15 AM [Nest] 1 - 01/17/2023, 8:49:15 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (1)... 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] AppModule dependencies initialized +1ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] JwtModule dependencies initialized +1ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] MailModule dependencies initialized +1ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] ConfigModule dependencies initialized +1ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] ConfigModule dependencies initialized +0ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] MailerCoreModule dependencies initialized +1ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] ServeStaticModule dependencies initialized +3ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] PassportModule dependencies initialized +1ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] MailerModule dependencies initialized +0ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [InstanceLoader] TypeOrmModule dependencies initialized +327ms 01-17-2023 09:49:05 AM [Nest] 1 - 01/17/2023, 8:49:05 AM  LOG [NestFactory] Starting Nest application... 01-17-2023 09:47:22 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:47:22 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:47:22 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:47:22 AM Error: connect ETIMEDOUT 01-17-2023 09:47:22 AM [Nest] 1 - 01/17/2023, 8:47:22 AM  ERROR [ExceptionHandler] connect ETIMEDOUT 01-17-2023 09:47:22 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:47:22 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:47:22 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:47:22 AM Error: connect ETIMEDOUT 01-17-2023 09:47:22 AM [Nest] 1 - 01/17/2023, 8:47:22 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (9)... 01-17-2023 09:47:09 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:47:09 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:47:09 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:47:09 AM Error: connect ETIMEDOUT 01-17-2023 09:47:09 AM [Nest] 1 - 01/17/2023, 8:47:09 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (8)... 01-17-2023 09:46:56 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:46:56 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:46:56 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:46:56 AM Error: connect ETIMEDOUT 01-17-2023 09:46:56 AM [Nest] 1 - 01/17/2023, 8:46:56 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (7)... 01-17-2023 09:46:43 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:46:43 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:46:43 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:46:43 AM Error: connect ETIMEDOUT 01-17-2023 09:46:43 AM [Nest] 1 - 01/17/2023, 8:46:43 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (6)... 01-17-2023 09:46:30 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:46:30 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:46:30 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:46:30 AM Error: connect ETIMEDOUT 01-17-2023 09:46:30 AM [Nest] 1 - 01/17/2023, 8:46:30 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (5)... 01-17-2023 09:46:17 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:46:17 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:46:17 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:46:17 AM Error: connect ETIMEDOUT 01-17-2023 09:46:17 AM [Nest] 1 - 01/17/2023, 8:46:17 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (4)... 01-17-2023 09:46:04 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:46:04 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:46:04 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:46:04 AM Error: connect ETIMEDOUT 01-17-2023 09:46:04 AM [Nest] 1 - 01/17/2023, 8:46:04 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (3)... 01-17-2023 09:45:51 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:45:51 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:45:51 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:45:51 AM Error: connect ETIMEDOUT 01-17-2023 09:45:51 AM [Nest] 1 - 01/17/2023, 8:45:51 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (2)... 01-17-2023 09:45:38 AM at processTimers (node:internal/timers:502:7) 01-17-2023 09:45:38 AM at listOnTimeout (node:internal/timers:559:17) 01-17-2023 09:45:38 AM at PoolConnection._handleTimeoutError (/usr/src/app/node_modules/mysql2/lib/connection.js:189:17) 01-17-2023 09:45:38 AM Error: connect ETIMEDOUT 01-17-2023 09:45:38 AM [Nest] 1 - 01/17/2023, 8:45:38 AM  ERROR [TypeOrmModule] Unable to connect to the database. Retrying (1)... 01-17-2023 09:45:29 AM [Nest] 1 - 01/17/2023, 8:45:29 AM  LOG [InstanceLoader] AppModule dependencies initialized +3ms 01-17-2023 09:45:29 AM [Nest] 1 - 01/17/2023, 8:45:29 AM  LOG [InstanceLoader] JwtModule dependencies initialized +78ms 01-17-2023 09:45:28 AM [Nest] 1 - 01/17/2023, 8:45:28 AM  LOG [InstanceLoader] MailModule dependencies initialized +5ms 01-17-2023 09:45:28 AM [Nest] 1 - 01/17/2023, 8:45:28 AM  LOG [InstanceLoader] ConfigModule dependencies initialized +0ms 01-17-2023 09:45:28 AM [Nest] 1 - 01/17/2023, 8:45:28 AM  LOG [InstanceLoader] ConfigModule dependencies initialized +1ms 01-17-2023 09:45:28 AM [Nest] 1 - 01/17/2023, 8:45:28 AM  LOG [InstanceLoader] MailerCoreModule dependencies initialized +1ms 01-17-2023 09:45:28 AM [Nest] 1 - 01/17/2023, 8:45:28 AM  LOG [InstanceLoader] ServeStaticModule dependencies initialized +74ms 01-17-2023 09:45:28 AM [Nest] 1 - 01/17/2023, 8:45:28 AM  LOG [InstanceLoader] ConfigHostModule dependencies initialized +3ms 01-17-2023 09:45:28 AM [Nest] 1 - 01/17/2023, 8:45:28 AM  LOG [InstanceLoader] PassportModule dependencies initialized +1ms 01-17-2023 09:45:28 AM [Nest] 1 - 01/17/2023, 8:45:28 AM  LOG [InstanceLoader] MailerModule dependencies initialized +14ms 01-17-2023 09:45:28 AM [Nest] 1 - 01/17/2023, 8:45:28 AM  LOG [InstanceLoader] TypeOrmModule dependencies initialized +103ms 01-17-2023 09:45:28 AM [Nest] 1 - 01/17/2023, 8:45:28 AM  LOG [NestFactory] Starting Nest application...

And here the yesterday RDS error log:

2023-01-17T09:36:47.703754Z 26135 [Warning] [MY-010056] [Server] Host name 'xxxxxx.6003333333.iuo' could not be resolved: Name or service not known 2023-01-17T11:14:48.656153Z 26155 [Warning] [MY-010055] [Server] IP address '71.6.232.24' could not be resolved: Name or service not known ----------------------- END OF LOG ----------------------

I confirm I have the same connection issue while using App Runner service with VPC connector.

What I really can't understand is why I get a connection timeout while I try to connect to the RDS DB when it's public and I can reach it without any issues from my office. 😅

Thanks in advance for your help.

helihang commented 1 year ago

Hi MrKappa,

The connection issue mostly related to permission to access the database. I tried multiple reproduces on connecting RDS to AppRunner service. Here are what I found when connection timeout happened and fixes.

  1. The security group attached to RDS does not have database port exposed. In my public accessible RDS, I also need to add the database port to be accessible as part of the inbound rule of security groups attached to the RDS. In the example I use mariadb with port 3306.
    Screen Shot 2023-01-18 at 5 11 49 PM
  2. Also another case when I remove the permission to connect to my RDS on the instance Role passed to App Runner service or detach the instance role from my service, the application also faced connection issues.

Can you double check on the both cases and let us know if these fix the problem. On the service side, App Runner service does not need extra configuration to connect to public accessible RDS. Here is my reproduced service with public accessible RDS. https://tmjtm7qekw.us-east-1.awsapprunner.com/ and the configurations.

Screen Shot 2023-01-18 at 5 32 32 PM

MrKappa commented 1 year ago

Hi helihang,

here my RDS sg: image

here my AppRunner security config: image

I'm still not able to get it working via AppRunner. Do you mind sharing your "secret-role"(AppRunner Instance role) config?

Thanks

lihanghe commented 1 year ago

Hi MrKappa,

Here is my instance role permissions, it allows the RDS connection to the database with all user and tables. The SecretsManagerReadWrite permission was added for my another App Runner service to access my secrets stores in AWS SecretsManager through environment variable, which should not be relevant to the issue you are seeing now.

Can you check if your instance role ("apprunner") has the corresponding permission to connect to RDS?

Screen Shot 2023-01-25 at 11 57 38 AM

MrKappa commented 1 year ago

Hi,

I have an equivalent instance role permission for accessing RDS. At this point I'll try to recreate everything from scratch and I'll keep you posted.

Thanks and sorry for my late reply.

amitgupta85 commented 1 year ago

Please keep us updated and let us know if this issue has been fixed for you.

janklan commented 1 year ago

@MrKappa How did you go?

MrKappa commented 1 year ago

I recreated everything from scratch (with CDK) and now it works. I still can't understand why it wasn't working before but I couldn't spend more time digging.

Sorry for my late reply and thank y'all for your help.