search

newrelic / micrometer-registry-newrelic

ARCHIVED. TO SEND MICROMETER METRICS TO NEW RELIC, FOLLOW THE DIRECTION IN THE README.md. Micrometer registry implementation that sends data to New Relic as dimensional metrics.
Apache License 2.0
35 stars 19 forks source link

Failed to send metrics to new relic SSLException: Received fatal alert: record_overflow #77

Closed alphafoobar closed 4 years ago

alphafoobar commented 4 years ago

We are using Spring boot and micrometer.io to publish metrics to New Relic Insights. Since the weekend ~12 April 2020, most of our services have been unable to publish any metrics. We had not updated any libraries or pushed any changes at the time - we saw an almost complete outage of all published metrics across most of our services, with no changes deployed at the time.

We are seeing a lot of errors like:

2020-04-14 17:29:11.157 WARN 20650 --- [ Thread-5] i.m.newrelic.NewRelicMeterRegistry : failed to send metrics to new relic

With a stack trace:

javax.net.ssl.SSLException: Received fatal alert: record_overflow
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:129)
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:308)
    at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:279)
    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:181)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1155)
    at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1125)
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)
    at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
    at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
    at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
    at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:746)
    at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
    at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:717)

It seems unlikely this is a configuration issue, or a code change since no changes have been deployed to our services over the weekend. And investigating the issue shows the problem is only intermittent. Running the same version of the application locally sometimes replicates the issue and sometimes it is able to publish metrics as designed.

Is anyone else experiencing this issue? The endpoint we are hitting is something like: https://insights-collector.newrelic.com/v1/accounts//events

We are using the following library to publish metrics group: 'io.micrometer', name: 'micrometer-registry-new-relic', version: '1.2.0'

jeffalder commented 4 years ago

if you have a service running on JDK 11.0.0 - 11.0.4 and you're seeing "TLS record_overflow" exceptions, it's most likely caused by this JDK bug https://bugs.openjdk.java.net/browse/JDK-8221253

Updating to 11.0.6 should fix it.

alphafoobar commented 4 years ago

I just realized this isn't the micrometer library we are using. But thanks a lot for the helpful suggestion!

alphafoobar commented 4 years ago

if you have a service running on JDK 11.0.0 - 11.0.4 and you're seeing "TLS record_overflow" exceptions, it's most likely caused by this JDK bug https://bugs.openjdk.java.net/browse/JDK-8221253

Updating to 11.0.6 should fix it.

This has fixed our issue. Thank you again.