Lambda RDS MySQL connection count issue

[sarkarstanmoy]

Hi, We are using Lambda's which are deployed in VPC to connect RDS instance which is again hosted in private subnet. We are using .Net Core 2.1, Pomelo.EntityFrameworkCore.MySql 2.2.4 version for Entity framework and MySql database. Currently, for one environment we are using 14 Lambda functions. The issue we are facing is RDS connection keeps on increasing and it went upto 70 connections for one user. If we stopped using lambda's then the connection count slowly coming down. To mitigate this issue we thought of using .Net core AddDbContextPool without much success. Need your suggestions on how to minimize the database connection count and improve performance.

[tsilopoulos]

Hi there,

It so happens that we have an identical configuration (EF Core 2.x via Pomelo connecting to Aurora MySQL RDS) in some of our own lambdas as the one you mention so here's one way you could introduce DbContextPool in each of your own lambdas; please note that the following snippet will need to be part of a method in the entry point of each lambda project (likely Function.cs) with the following method signature ConfigureServices(IServiceCollection services) or similar.

services.AddDbContextPool<VidreachDbContext>(
    options => options.UseMySql(auroraConnectionString,
    mysqlOptions =>
    {
        mysqlOptions.ServerVersion(new Version(5, 7, 12), ServerType.MySql);
    }
));

[sarkarstanmoy]

I have already tried this by including into startup.cs file with no success. By default adddbcontextpool is 128 pool size. So everytime lambda spawns it is creating new dbconnection object and count getting up. I have seen for node.js people are suggesting to create connection globally so that it can be shared among lambdas. I am not sure how to do it in .net core. Please let me know any suggestions.

[tsilopoulos]

What does "no success" mean in this context?

Is the lambda successfully invoked and execute to completion after you added the DbContextPool but the number of open connections to your database is still roughly the same? Why would you expect that figure to be different?

Unless I'm mistaken and I could very well be, the connection pool is simply reusing existing connections to the database so the difference should be that by using DbContextPool the number of active connections from any one lambda invocation to your MySQL database should, at worse, be equal to that of the value of the pool size.

What you gain by using the DbContextPool is that it won't ever exceed that figure for any single lambda invocation.

If you still have a problem with the number of active connections and you have multiple lambdas running at the same time, perhaps it's time to consider either limiting the connection pool size per function or removing concurrency from those lambdas (aka figure out a way to prevent multiple invocations of each of those, always run 1 or N at one time).

[sarkarstanmoy]

You are right that connection pooling should reuse the existing connection in database. However, in case of lambda (serverless) it is not able to share the dbcontextpool and for each request new lambda instance is getting spawn up. For e.g. for N invocation N lambdas started. Since all are new lambdas instance and have there own execution context they are not able to share the dbconnection.

However, i have limit the concurrency execution of each lambda which in fact arrest the db connection count issue. But now if there are multiple calls for lambda and no instances are free then it is throwing 500 which is expected as per Amazon documentation. E.g. if I limit concurrency execution to 10 and there are 20 request at the same time 20-10=10 requests are throwing 500 until they have been processed by lambda. How to overcome this situation?

[Kralizek]

This is a general problem when at the boundary between the serverless (almost infinite scalability) and the serverful (finite scalability) worlds.

Depending on your problem, you might want to consider the possibility to use a queue to buffer those access to the database so that the database is not under the threat of an explosion of requests from lambda.

[tsilopoulos]

There are two ways you could control the concurrency of those lambdas:

• either use Step Functions where you might define the granularity of such operations or
• use a cache (like AWS Elasticache) to control not how many instances of each lambda are concurrently invoked but rather which of them make it past a certain "choke point" (where you'd be checking against the cache in your code) and in your case establish a connection to the database.

The forrmer implementation would both require time as well as a rather steep learning curve involved for your developers so if time is of essence, I would recommend the second route.

[klaytaybai]

Hi everyone, thanks for the community involvement with this issue. Those seems like they could be helpful solutions. There are plenty of other reading materials around this topic online from a variety of sources. I suggest you continue reading and find a strategy that works best for you. I'm going to close this issue because it isn't really an issue with the .NET SDK. Feel free to keep commenting here for now, but I also suggest commenting on existing issues in the AWS Forums, Stack Overflow, and other sites discussing this topic.