Honeycomb opentelemetry metrics application hangs on exit

smahesh-panw commented 2 years ago

I am trying out the open-telemetry metrics feature with opentelemetry-java version v1.11.0. I need to use the capture interval=120secs and metrics temporality=DELTA settings.

From my experiments, it looks like only opentelemetry-javaagent v1.11.0 works with opentelemetry-java v1.11.0.

Using latest honeycomb-javaagent (v0.9.0) causes the application to not exit (main thread terminates, but the BatchSpanProcessor thread never gets the shutdown signal).

The honeycomb-agent v0.9.0 is not compatible with opentelemetry-java v1.9.1 either - the application shows similar hang-on-exit behavior.

Slack Ref: https://honeycombpollinators.slack.com/archives/C4T3A32CX/p1646042652657739

Versions Opentelemetry v1.11.0 and v1.9.2 Honeycomb agent v0.9.0

Java: 11.0
Honeycomb OpenTelemetry Distribution: v0.9.0

Steps to reproduce

Run the attached sample code. It emits metrics every 10 seconds. Application generates metrics every 20 seconds. Metrics should be available every alternate export cycle.
With honeycomb javaagent 0.9.0, the application hangs-on-exit. The main thread terminates, but the BatchSpanProcessor does not get shutdown signal - the metric emitter thread does not shutdown and the JVM does not exit.
The sample code was tested with opentelemetry v1.11.0 and v1.9.2 - both exhibit the hang-on-exit behavior. The onloy working solution is to use opentelemetry javaagent v1.11.0 with opentelemetry-java v1.11.0.

package com.paloaltonetworks;

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.metrics.LongHistogram;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.api.trace.Tracer;

import java.util.ArrayList;
import java.util.List;

/**
 * Sample code for Honeycomb opentelemetry traces and metrics
 *
 */
class HoneyCombSample {
  private Tracer tracer = GlobalOpenTelemetry.getTracer("tracer");
  private Meter meter = GlobalOpenTelemetry.getMeter("HoneyCombSample");
  private final LongHistogram incomingRecordsCounter = meter.histogramBuilder("incomingRecordsCounter").ofLongs().build();

  class Record {
    private long customerId;
    private long value;

    public Record(final long customerId, final long value) {
      this.customerId = customerId;
      this.value = value;
    }

    @Override
    public String toString() {
      return "Record{customerId=" + customerId + ", value=" + value + '}';
    }
  }

  private List<Record> getRecords() {
    //NOTE: There are duplicate records - needed to send duplicate metrics
    List<Record> records = new ArrayList<>(10);
    records.add(new Record(1, 1001));
    records.add(new Record(2, 1002));
    records.add(new Record(3, 1003));
    records.add(new Record(4, 1004));
    records.add(new Record(5, 1005));
    records.add(new Record(1, 2001));
    records.add(new Record(2, 2002));
    records.add(new Record(3, 2003));
    records.add(new Record(1, 3001));
    records.add(new Record(2, 3002));
    return records;
  }

  public void processRecords(final int iteration) {
    List<Record> records = getRecords();

    for(Record record: records) {
      incomingRecordsCounter.record(1, Attributes.of(AttributeKey.longKey("record.customerId"), record.customerId));
    }
  }

  public static void main(String[] args) {
    HoneyCombSample hcs = new HoneyCombSample();
    for (int i = 0; i < 5; i++) {
      System.out.println("Iter ... " + i);
      hcs.processRecords(i+1);
      try {
        System.out.println("Sleeping ....");
        // Sleep for double the OTEL export interval - metrics should be emitted every other cycle
        Thread.sleep(20*1000);
      } catch (InterruptedException e) {
        e.printStackTrace();
      }
    }
    System.out.println("DONE....");
  }
}

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.paloaltonetworks</groupId>
  <artifactId>HoneycombSample</artifactId>
  <version>1.0-SNAPSHOT</version>

  <name>HoneycombSample</name>
  <url>https://www.paloaltonetworks.com</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>11</maven.compiler.source>
    <maven.compiler.target>11</maven.compiler.target>
  </properties>
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-bom</artifactId>
        <version>1.11.0</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>

      <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-bom-alpha</artifactId>
        <version>1.11.0-alpha</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-api</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-api</artifactId>
    </dependency>

    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-sdk-metrics</artifactId>
    </dependency>

    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-sdk</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-sdk-metrics</artifactId>
    </dependency>

    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-sdk</artifactId>
    </dependency>
  </dependencies>

  <build>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
      <plugins>
        <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
        <plugin>
          <artifactId>maven-clean-plugin</artifactId>
          <version>3.1.0</version>
        </plugin>
        <!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
        <plugin>
          <artifactId>maven-resources-plugin</artifactId>
          <version>3.0.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.8.0</version>
        </plugin>
        <plugin>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.22.1</version>
        </plugin>
        <plugin>
          <artifactId>maven-jar-plugin</artifactId>
          <version>3.0.2</version>
          <configuration>
            <excludes>**/*.avsc</excludes>
          </configuration>
        </plugin>
        <plugin>
          <artifactId>maven-install-plugin</artifactId>
          <version>2.5.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-deploy-plugin</artifactId>
          <version>2.8.2</version>
        </plugin>
        <!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
        <plugin>
          <artifactId>maven-site-plugin</artifactId>
          <version>3.7.1</version>
        </plugin>
        <plugin>
          <artifactId>maven-project-info-reports-plugin</artifactId>
          <version>3.0.0</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</project>

Additional context Set the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team\=<API_KEY>,x-honeycomb-dataset\=honeycomb-demo
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://api.honeycomb.io:443
OTEL_EXPORTER_OTLP_METRICS_HEADERS=x-honeycomb-team\=<API_KEY>,x-honeycomb-dataset\=honeycomb-demo-metrics
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY=DELTA
OTEL_INSTRUMENTATION_KAFKA_ENABLED=false
OTEL_METRICS_EXPORTER=otlp
OTEL_METRIC_EXPORT_INTERVAL=10000
OTEL_SERVICE_NAME=HoneycombSample
OTEL_TRACES_EXPORTER=otlp
HONEYCOMB_API_KEY=<API_KEY>
HONEYCOMB_DATASET=honeycomb-demo
SERVICE_NAME=HoneycombSample
HONEYCOMB_METRICS_DATASET=honeycomb-demo-metrics
HONEYCOMB_METRICS_APIKEY=<API_KEY>

smahesh-panw commented 2 years ago

This is what visualvm reports when the application hangs

MikeGoldsmith commented 2 years ago

Hi @smahesh-panw - we released v0.10.0 of the honeycomb java distro yesterday which includes upgrading to the latest version of OpenTelemetry SDK (v1.11.0) & Agent (v1.11.1).

Please try out the latest honeycomb distro and let us know if this resolves your issue.

PS It is expected that the OpenTelemetry Agent is built against a particular version of the SDK and they must match otherwise you'll see inconsistent behaviour.

smahesh-panw commented 2 years ago

Hi Mike,

Honeycomb javaagent v0.10.0 works correctly with open-telemetry v1.11.0. The app does not hang on exit anymore.

Regards, Mahesh

On Thu, Mar 3, 2022 at 4:12 PM Mike Goldsmith @.***> wrote:

Hi @smahesh-panw https://urldefense.com/v3/__https://github.com/smahesh-panw__;!!Mt_FR42WkD9csi9Y!IdFJECGJko14iV2oncX04rtp6WkccEBuBtqPakoRoiUa0qUKpMqAXj-tqq1Xe6rmzTm7DA$

we released v0.10.0 https://urldefense.com/v3/__https://github.com/honeycombio/honeycomb-opentelemetry-java/releases/tag/v0.10.0__;!!Mt_FR42WkD9csi9Y!IdFJECGJko14iV2oncX04rtp6WkccEBuBtqPakoRoiUa0qUKpMqAXj-tqq1Xe6r8yP4rZQ$ of the honeycomb java distro yesterday which includes upgrading to the latest version of OpenTelemetry SDK (v1.11.0) & Agent (v1.11.1).

Please try out the latest honeycomb distro and let us know if this resolves your issue.

PS It is expected that the OpenTelemetry Agent is built against a particular version of the SDK and they must match otherwise you'll see inconsistent behaviour.

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/honeycombio/honeycomb-opentelemetry-java/issues/270*issuecomment-1057912232__;Iw!!Mt_FR42WkD9csi9Y!IdFJECGJko14iV2oncX04rtp6WkccEBuBtqPakoRoiUa0qUKpMqAXj-tqq1Xe6rtAYUVAA$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AYBGMWUBBW3Q73NFCHWDMS3U6CJRLANCNFSM5PZKKQ7Q__;!!Mt_FR42WkD9csi9Y!IdFJECGJko14iV2oncX04rtp6WkccEBuBtqPakoRoiUa0qUKpMqAXj-tqq1Xe6qk7kuC-g$ . Triage notifications on the go with GitHub Mobile for iOS https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!Mt_FR42WkD9csi9Y!IdFJECGJko14iV2oncX04rtp6WkccEBuBtqPakoRoiUa0qUKpMqAXj-tqq1Xe6riBTJv0w$ or Android https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!Mt_FR42WkD9csi9Y!IdFJECGJko14iV2oncX04rtp6WkccEBuBtqPakoRoiUa0qUKpMqAXj-tqq1Xe6oHoGhSnQ$.

You are receiving this because you were mentioned.Message ID: @.*** com>

MikeGoldsmith commented 2 years ago

Thanks for confirmin 👍🏻

honeycombio / honeycomb-opentelemetry-java

Honeycomb opentelemetry metrics application hangs on exit #270