Different in size of data volume in ServiceNow Cloud Observability and Azure Application Insight logs from OTel Collector chart

[sahanruwanga]

Description

I'm using opentelemetry-collector helm chart in an AKS cluster to receive and export logs, metrics, and traces from AKS workloads. There are two exporters. One is an Azure Application Insight resource and the other one is Service Now Cloud Observability(lightstep).

Even though I expect that I will have same data(volume) in both Azure Application Insight and Service Now Cloud Observability, I can see 3~4 times data volume in Azure Application Insight compared to what I see in SN cloud observability. I'm trying to understand the reason for this difference. Can it be depended on the platform/service that I config as exporter endpoints and how/what they handle/store data? Any supportive answer would be highly appreciated.

opentelemetry-helm-charts version: 0.93.3

Following is the related configuration in my open-telemetry-opentelemetry-collector-agent config map in AKS.

exporters:
      azuremonitor/appi:
        connection_string: <value>
      otlp/cloud-obs:
        endpoint: ingest.lightstep.com:443
        headers:
          lightstep-access-token: <value>

receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

service:
      pipelines:
        logs:
          exporters:
          - azuremonitor/appi
          - otlp/cloud-obs
          receivers:
          - otlp
        metrics:
          exporters:
          - azuremonitor/appi
          - otlp/cloud-obs
          receivers:
          - otlp
        traces:
          exporters:
          - azuremonitor/appi
          - otlp/cloud-ob
          receivers:
          - otlp

[dosubot[bot]]

Hello, @sahanruwanga! I'm here to assist you with any questions or issues you may have. Let's work together to resolve your concern.

The difference in data volume between Azure Application Insights and ServiceNow Cloud Observability could be due to several factors, including how each platform processes, stores, and compresses the data.

For the azuremonitor/appi exporter, there are no specific known issues or limitations documented that could lead to higher data volumes . Additionally, there are no specific configuration options or parameters for the azuremonitor/appi exporter that are known to influence the volume of data being sent .

However, it's possible that the platforms have different data retention policies, compression algorithms, or data enrichment processes that could affect the volume of data stored. You may want to review the documentation for both Azure Application Insights and ServiceNow Cloud Observability to understand how they handle incoming telemetry data.

To continue talking to Dosu, mention @dosu.

[jaronoff97]

how is data volume measured in azure? It's possible that the numbers you're seeing are different between platforms based on storage and other potential sources. I would query the opentelemetry collector metrics to confirm that data is successfully being exported to each platform. I would also look at the collector logs and see if there are any issues there. For now, I don't think this is a helm bug and will close this issue.