search

newrelic / newrelic-java-agent

The New Relic Java agent
Apache License 2.0
198 stars 142 forks source link

Investigate potential high memory usage with r2dbc-mssql/r2dbc-mysql instrumentation #2015

Open jasonjkeller opened 3 weeks ago

jasonjkeller commented 3 weeks ago

Related to https://new-relic.atlassian.net/browse/NR-299687

Customer reported that they were getting an OOM with their service using SpringBoot and R2DBC Mysql.

It was confirmed that disabling the r2dbc-mysql instrumentation resolved the memory issues:

  class_transformer:
    com.newrelic.instrumentation.r2dbc-mysql:
      enabled: false

This may be related to the the use of the @NewField annotation and a known issue with how they interact with the caffeine cache when there are many instances of the annotation.

https://github.com/newrelic/newrelic-java-agent/blob/d35ffd63177c2bffac036b1be5fec9fef6f420c1/instrumentation/r2dbc-mssql/src/main/java/io/r2dbc/mssql/client/ReactorNettyClient_Instrumentation.java#L8-L16

https://github.com/newrelic/newrelic-java-agent/blob/d35ffd63177c2bffac036b1be5fec9fef6f420c1/instrumentation/r2dbc-mysql/src/main/java/dev/miku/r2dbc/mysql/client/ReactorNettyClient_Instrumentation.java#L10-L18

It may also be the underlying netty-reactor instrumentation that is the real issue as the heap dump shows a lot of cached entries for reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.

Screenshot 2024-08-19 at 8 56 55 AM

Here is the instrumentation that was loaded in the environment that was experiencing the OOM:

com.newrelic.instrumentation.rabbit-amqp-5.0.0
com.newrelic.instrumentation.jdbc-driver
com.newrelic.instrumentation.spring-webclient-5.0
com.newrelic.instrumentation.netty-4.1.16
com.newrelic.instrumentation.netty-reactor-0.9.0
com.newrelic.instrumentation.httpurlconnection
com.newrelic.instrumentation.java.logging-jdk8
com.newrelic.instrumentation.logback-classic-1.2
com.newrelic.instrumentation.jdbc-mysql-connection-8.0.11
com.newrelic.instrumentation.jdbc-mysql-8.0.11
com.newrelic.instrumentation.spring-cache-3.1.0
com.newrelic.instrumentation.r2dbc-mysql
com.newrelic.instrumentation.java.process
com.newrelic.instrumentation.jdbc-socket
com.newrelic.instrumentation.kafka-clients-spans-0.11.0.0
com.newrelic.instrumentation.spring-webflux-controller-mappings-5.0.0
com.newrelic.instrumentation.netty-4.0.8
com.newrelic.instrumentation.java.completable-future-jdk11
com.newrelic.instrumentation.java.completable-future-jdk8u40
com.newrelic.instrumentation.httpclient-4.0
com.newrelic.instrumentation.spring-webflux-5.3.0
com.newrelic.instrumentation.jdbc-generic
com.newrelic.instrumentation.kafka-clients-metrics-3.0.0
workato-integration[bot] commented 3 weeks ago

https://new-relic.atlassian.net/browse/NR-304225