Memory leak in efm2 plugin

KrohnicDev commented 7 months ago

Describe the bug

We have noticed out of memory errors, and based on heap dump analysis it is due to efm2 plugin.

Also, we have set wrapperPlugins to emf,failover so we were quite suprised to see efm2 as the root cause of this problem. efm classes are not visible in heap dump, so it seems that efm2 is used despite of configuration.

Expected Behavior

Wrapper should work without memory leaks.

What plugins are used? What other connection properties were set?

Properties extracted from heap dump:

targetDataSourceProperties = {
  failureDetectionEnabled=true, 
  wrapperDialect=aurora-pg, 
  wrapperPlugins=efm,failover, 
  failureDetectionInterval=5000, 
  failureDetectionCount=3, 
  failureDetectionTime=30000, 
  monitorDisposalTime=60000
}

Current Behavior

Our application crashes to OOM errors. Based on heap dump analysis it is due to efm2 plugin, which has a memory leak in newContexts:

Sounds a lot like https://github.com/awslabs/aws-advanced-jdbc-wrapper/issues/675, but different plugin.

Reproduction Steps

I don't have the exact reproduction steps.

Possible Solution

No response

Additional Information/Context

The memory leak was for the readonly connection pool (targeting RDS read endpoint). That's probably relevant, since it's also the only pool that gets these errors:

java.util.ConcurrentModificationException: null
    at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
    at software.amazon.jdbc.plugin.efm2.MonitorImpl.startMonitoring(MonitorImpl.java:169)
    at software.amazon.jdbc.plugin.efm2.MonitorServiceImpl.startMonitoring(MonitorServiceImpl.java:118)
    at software.amazon.jdbc.plugin.efm2.HostMonitoringConnectionPlugin.execute(HostMonitoringConnectionPlugin.java:169)
    at software.amazon.jdbc.ConnectionPluginManager.lambda$execute$5(ConnectionPluginManager.java:323)
    at software.amazon.jdbc.ConnectionPluginManager.lambda$null$3(ConnectionPluginManager.java:260)
    at software.amazon.jdbc.ConnectionPluginManager.executeWithTelemetry(ConnectionPluginManager.java:233)
    at software.amazon.jdbc.ConnectionPluginManager.lambda$makePluginChainFunc$4(ConnectionPluginManager.java:260)
    at software.amazon.jdbc.ConnectionPluginManager.lambda$null$2(ConnectionPluginManager.java:261)
    at software.amazon.jdbc.plugin.failover.FailoverConnectionPlugin.execute(FailoverConnectionPlugin.java:225)

and warnings from HikariCP:

ReadOnlyPool - Failed to validate connection software.amazon.jdbc.wrapper.ConnectionWrapper@67e56e83 - 
org.postgresql.jdbc.PgConnection@1beb7df6 (java.util.ConcurrentModificationException). 
Possibly consider using a shorter maxLifetime value.

The AWS Advanced JDBC Driver version used

2.3.3

JDK version used

21

Operating System and version

AWS ECS Fargate

sergiyvamz commented 7 months ago

Hi @KrohnicDev

The issue has been fixed in version 2.3.4. We suggest you to upgrade to the latest version of the driver (that is version 2.3.5). https://github.com/awslabs/aws-advanced-jdbc-wrapper/issues/855

Thank you!

KrohnicDev commented 7 months ago

Thanks! We had to skip v2.3.4 due another issue, but v2.3.5 seems to work fine.

aws / aws-advanced-jdbc-wrapper