Exporter is unable to stop retrying for connection during a fatal exception

[spatanjali]

The exporter is unable to stop retrying while facing an exception when it tries to send a Batchdata. This exception is observed during the application startup and is blocking the application from starting up.

Version: 0.6.0 Spring Boot: 2.3.5.RELEASE

Stack Trace `2020-11-13T14:01:07,534 ERROR [Thread-19] c.n.t.TelemetryClient: Received a fatal exception from the New Relic API. Aborting metric batch send. com.newrelic.telemetry.exceptions.DiscardBatchException: The New Relic API failed to process this request and it should not be retried. at com.newrelic.telemetry.transport.BatchDataSender.sendPayload(BatchDataSender.java:138) at com.newrelic.telemetry.transport.BatchDataSender.send(BatchDataSender.java:82) at com.newrelic.telemetry.metrics.MetricBatchSender.sendBatch(MetricBatchSender.java:74) at com.newrelic.telemetry.TelemetryClient.lambda$sendBatch$0(TelemetryClient.java:127) at com.newrelic.telemetry.TelemetryClient.sendWithErrorHandling(TelemetryClient.java:191) at com.newrelic.telemetry.TelemetryClient.lambda$scheduleBatchSend$4(TelemetryClient.java:182) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

2020-11-13T14:01:12,523 ERROR [Thread-19] c.n.t.TelemetryClient: Received a fatal exception from the New Relic API. Aborting metric batch send. com.newrelic.telemetry.exceptions.DiscardBatchException: The New Relic API failed to process this request and it should not be retried. at com.newrelic.telemetry.transport.BatchDataSender.sendPayload(BatchDataSender.java:138) at com.newrelic.telemetry.transport.BatchDataSender.send(BatchDataSender.java:82) at com.newrelic.telemetry.metrics.MetricBatchSender.sendBatch(MetricBatchSender.java:74) at com.newrelic.telemetry.TelemetryClient.lambda$sendBatch$0(TelemetryClient.java:127) at com.newrelic.telemetry.TelemetryClient.sendWithErrorHandling(TelemetryClient.java:191) at com.newrelic.telemetry.TelemetryClient.lambda$scheduleBatchSend$4(TelemetryClient.java:182) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

2020-11-13T14:01:14,976 ERROR [Thread-19] c.n.t.TelemetryClient: Received a fatal exception from the New Relic API. Aborting metric batch send. com.newrelic.telemetry.exceptions.DiscardBatchException: The New Relic API failed to process this request and it should not be retried. at com.newrelic.telemetry.transport.BatchDataSender.sendPayload(BatchDataSender.java:138) at com.newrelic.telemetry.transport.BatchDataSender.send(BatchDataSender.java:82) at com.newrelic.telemetry.metrics.MetricBatchSender.sendBatch(MetricBatchSender.java:74) at com.newrelic.telemetry.TelemetryClient.lambda$sendBatch$0(TelemetryClient.java:127) at com.newrelic.telemetry.TelemetryClient.sendWithErrorHandling(TelemetryClient.java:191) at com.newrelic.telemetry.TelemetryClient.lambda$scheduleBatchSend$4(TelemetryClient.java:182) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)`

Configuration `@Configuration @AutoConfigureBefore({ CompositeMeterRegistryAutoConfiguration.class, SimpleMetricsExportAutoConfiguration.class }) @AutoConfigureAfter(MetricsAutoConfiguration.class) @ConditionalOnClass(NewRelicRegistry.class) public class MicrometerConfig {

@Value("${newrelic.insert.api.key}")
private String nrInsertApiKey;

@Value("${newrelic.metric.uri}")
private String nrMetricUri;

@Value("${nr.application.name}")
String serviceName;

@Bean
public NewRelicRegistryConfig newRelicConfig() {

    return new NewRelicRegistryConfig() {
        @Override
        public String get(String key) {
            return null;
        }

        @Override
        public String apiKey() {
            return nrInsertApiKey;
        }

        @Override
        public String uri() {
            return nrMetricUri;
        }

        @Override
        public Duration step() {
            return Duration.ofSeconds(5);
        }

        @Override
        public String serviceName() {
            return serviceName;
        }
    };
}

@Bean
public NewRelicRegistry newRelicMeterRegistry(NewRelicRegistryConfig config)
    throws UnknownHostException {
    NewRelicRegistry newRelicRegistry =
        NewRelicRegistry.builder(config)
            .commonAttributes(
                new Attributes()
                    .put("host.hostname", InetAddress.getLocalHost().getHostName()))
            .build();
    newRelicRegistry.start(new NamedThreadFactory("newrelic.micrometer.registry"));
    return newRelicRegistry;
}

}`

[breedx-nr]

Thanks @spatanjali for bringing this to our attention. Do you know of any reasons why the data are being rejected? Did something change to break this, or is this your first attempt at using the registry?

I think it's a little odd that the TelemetryClient failures would block application startup...they're supposed to be happening on a background thread.

Is there any chance you could link to an application that reproduces this behavior?

[kford-newrelic]

@spatanjali Thanks for letting us know about this issue. If you're ever able to repro this with a small sample application that we can take a look at, it would be appreciated.