Traces exported via otlp to Instana can not be viewable on Instana dashboard if metricstransform processor is involved

[xwgao]

Component(s)

processor/metricstransform

What happened?

Description

I enabled auto-instrumentation for my microservices in Node.js. And I used spanmetrics connector and metricstransform processor in my otel collector to generate metrics (duration and error) for microservices in my application. And I also configured my otel collector to export traces and metrics to Instana server via OTLP protocol. If I added the below transformation in metricstransform processor as below, I would not see the traces for endpoint /api/v2/document-query/query on Instana dashboard. After removing this transformation, I can see the traces for endpoint /api/v2/document-query/query on Instana dashboard.

metricstransform:
        transforms:
          ...
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "http.target": "/api/v2/document-query/query$",
              "method": "GET"
            }
            new_name: mas-assist.documentquery_searchdocuments_duration_seconds

Steps to Reproduce

See the details in ##Description

Expected Result

I can see the traces for endpoint /api/v2/document-query/query on Instana dashboard even if I added the following transformation in metricstransform processor as below.

metricstransform:
        transforms:
          ...
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "http.target": "/api/v2/document-query/query$",
              "method": "GET"
            }
            new_name: mas-assist.documentquery_searchdocuments_duration_seconds

Actual Result

I can not see the traces for endpoint /api/v2/document-query/query on Instana dashboard if I added the following transformation in metricstransform processor as below.

metricstransform:
        transforms:
          ...
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "http.target": "/api/v2/document-query/query$",
              "method": "GET"
            }
            new_name: mas-assist.documentquery_searchdocuments_duration_seconds

Collector version

v0.75.0

Environment information

Environment

OpenShift Container Platform 4.10

OpenTelemetry Collector configuration

spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      otlp/spanmetrics:
        protocols:
          grpc:
            endpoint: 0.0.0.0:12346

    processors:
      batch:
        timeout: 10s
        send_batch_size: 10000
      spanmetrics:
        metrics_exporter: prometheus
        latency_histogram_buckets:
          [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 5s, 7s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.target
          - name: http.url
      metricstransform:
        transforms:
          - include: mas-assist.duration
            match_type: regexp
            action: update
            operations:
              - action: add_label
                new_label: masinstanceid
                new_value: assistdevtest
              - action: add_label
                new_label: masappid
                new_value: assist
              - action: add_label
                new_label: masworkspaceid
                new_value: masdev
              - action: update_label
                label: http.url
                new_label: url
              - action: update_label
                label: http.method
                new_label: method
              - action: update_label
                label: http.status_code
                new_label: code
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "http.target": "/api/v1/session-management/sessionsCount",
              "method": "GET"
            }
            new_name: mas-assist.collaborate_getsessioncount_duration_seconds
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "http.target": "/api/v2/document-query/query$",
              "method": "GET"
            }
            new_name: mas-assist.documentquery_searchdocuments_duration_seconds

    exporters:
      otlp:
        endpoint: otlp-orange-saas.instana.io:4317
        headers:
          x-instana-key: uBp4GXpZQpKrHxMXNcvInQ
          x-instana-host: 0.0.0.0
      logging:
        verbosity: detailed
      jaeger:
        endpoint: my-jaeger-collector-headless.jaeger-demo.svc:14250
        tls:
          ca_file: "/etc/pki/ca-trust/source/service-ca/service-ca.crt"
      prometheus:
        endpoint: "0.0.0.0:8889"
        send_timestamps: true
        metric_expiration: 1440m

    connectors:
      spanmetrics:
        namespace: mas-assist
        histogram:
          unit: s
          explicit:
            buckets: [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 5s, 7s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.target
          - name: http.url
          - name: http.route

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [spanmetrics, logging, jaeger, otlp]
        metrics:
          receivers: [otlp/spanmetrics, spanmetrics]
          processors: [batch, metricstransform]
          exporters: [prometheus, logging, otlp]
  mode: statefulset
  volumeMounts:
    - mountPath: /etc/pki/ca-trust/source/service-ca
      name: cabundle-volume
  volumes:
    - configMap:
        name: my-otelcol-cabundle
      name: cabundle-volume
  image: >-
    ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.75.0

### Log output

_No response_

### Additional context

_No response_

[github-actions[bot]]

Pinging code owners:

• processor/metricstransform: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/metricstransform: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

This issue has been closed as inactive because it has been stale for 120 days with no activity.