Migrating to OpenTelemetry Distro?

[nmakanderhitatchi]

I've been trying to migrate from OpenTelemetry Distro from App Insights 2.9.4 but I am not getting the backend services to show up in the Application Map or any requests in Live Metrics. There's no correlation between our frontend and our backend. Neither with AppInsights 3.3.x or the @azure/monitor-opentelemetry 1.7.1.

This is running inside a node22 alpine container with "express": "^4.21.0".

With version 2.9.x I am using a fairly simple setup:

const appInsights = require('applicationinsights');

    appInsights
      .setup('connectionstring')
      .setAutoCollectConsole(true, true)
      .setSendLiveMetrics(true);

but using a similar basic setup with the two newer versions does not work. I've followed the offical documentation on the support pages but am not getting any results from that.

The only part that seems to work with the OpenTelemetry Distro is the ability to name the service using SemanticResourceAttributes.SERVICE_NAMESPACE. However, neither version 3 nor the OpenTelemetry Distro appear in the Application Map or register in Live Metrics aside from CPU usage.

Do you have any suggestions?

[JacksonWeber]

Hi @nmakanderhitatchi can I take a look at your SDK initialization for AppInsights 3.3.x?

[nmakanderhitatchi]

@JacksonWeber it's been setup the same way, just a change in package version.

I find this to be a bit confusing as well: https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-nodejs-migrate?tabs=upgrade

The way I read it here is that there's just a version upgrade and it should work as it did before but there are some properties that have changed that are located here: Unsupported Properties.

When I go to that page and look scroll up it seems like it's being initialized with

useAzureMonitor(config);

I expected it to be the same setup as with version 2 considering upgrade instructions said it was just npm update application insights.

I did try to set it up with the useAzureMonitor as well though but I didn't get the metrics to show up aside from CPU, no traces or anything visible in the Application Map.

 const customResource = new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: 'my-service',
      [SemanticResourceAttributes.SERVICE_NAMESPACE]: 'my-namespace',
      [SemanticResourceAttributes.SERVICE_INSTANCE_ID]: 'my-instance',
    });

    const options = {
      // Sampling could be configured here
      samplingRatio: 1,
      // Use custom Resource
      resource: customResource,
      azureMonitorExporterOptions: {
        connectionString: "connection string"
      },
      instrumentationOptions: {
        // Custom HTTP Instrumentation Configuration
        azureSdk: { enabled: true },
        mongoDb: { enabled: true },
        mySql: { enabled: true },
        postgreSql: { enabled: true },
        redis: { enabled: true },
        redis4: { enabled: true },
      },
    };

    useAzureMonitor(options);

[JacksonWeber]

I see the confusion here. That link to the unsupported properties is taking you to the beta branch of the Application Insights project, and while the unsupported properties listed there are correct, the information above about initialization of the SDK will not match that of the main branch which is what should be followed for initializing the 3.x SDK. I'll ensure that link gets updated to the correct README.

Given that, you are correct that the setup should be the same as version 2.x. However, I'm curious which product suits your use-case best (do you have an existing instrumentation using the 2.x SDK, are you trying to transition to taking advantage of the Azure Monitor OpenTelemetry Distro). Initially you mentioned that you were trying to migrate to the distro but were encountering this issue, so it sounds like you transitioned to trying to use the 3.x SDK instead. I just want to focus on one product/initialization so I can isolate your issue. Thanks!

[nmakanderhitatchi]

@JacksonWeber, thanks! We're currently instrumenting with 2.x and that works fine for us. However we like staying on top of things so we're looking for the latest version that provides the same features as version 2.x, particularly live metrics, tracing, and the application map. The simplicity of configuration is also important; it was great to have a one-liner setup with App Insights 2.x. We started experimenting with OpenTelemetry but couldn't get the Application Map or Live Metrics working, so we switched to version 3 instead but got stuck there too.

[JacksonWeber]

@nmakanderhitatchi So long as the features you utilize aren't among those in the unsupported properties list, 3.x should work great for you. If you do want to get the latest and greatest with the benefits of OpenTelemetry, that will always be available in the Azure Monitor OpenTelemetry distro.

As for the issue you're experiencing, I'll start with some steps we can try with this project (Application Insights v3 SDK). You've mentioned that you're not receiving requests in live metrics, but I wanted to confirm that you are still receiving telemetry of some kind? You also mentioned that in some cases the CPU usage metric is coming through in Live Metrics, would it be possible to get a screenshot of what the live metrics blade looks like for you when you've instrumented with the v3 SDK?

[nmakanderhitatchi]

@JacksonWeber Sounds like we v3 will be a good fit for now. This is what I'm seeing in the live metrics tab with version 3 and our frontend client doesn't appear in the application map anymore.

   exitOnError: true,
   silent: false,
   level: 'debug',
   azureAiConnectionString: 'some connection string',
   azureRoleName: 'APP_INSIGHTS_V3_DS',
   azureSendLiveMetrics: true,
   autoDependencyCorrelation: true,
   autoCollectRequests: true,
   autoCollectPerformance: true,
   autoCollectExceptions: true,
   autoCollectDependencies: true,
   autoCollectConsole: true,
   useDiskRetryCaching: true,
   distributedTracingMode: 'W3C_AND_AI',
   samplingPercentage: 33
 }
 @azure/opentelemetry-instrumentation-azure-sdk [
   'Module @azure/core-tracing has been loaded before @azure/opentelemetry-instrumentation-azure-sdk so it might not work, please initialize it before requiring @azure/core-tracing'
 ]
 @opentelemetry/instrumentation-winston [
   'Module winston has been loaded before @opentelemetry/instrumentation-winston so it might not work, please initialize it before requiring winston'
 ]
 @opentelemetry/instrumentation-winston [
   'Module winston has been loaded before @opentelemetry/instrumentation-winston so it might not work, please initialize it before requiring winston'
 ]
 The 'logRecordProcessor' option is deprecated. Please use 'logRecordProcessors' instead. []
 Extended metrics are no longer supported. Please reference the Azure Monitor OpenTelemetry Migration Doc for more information. If this functionality is required, please revert to Application Insights 2.X SDK. []
 We are running app insights v3!

[JacksonWeber]

@nmakanderhitatchi Live metrics blade looks to be reporting correctly. You mentioned that there might be an issue with context correlation given that you're not seeing your frontend service reported from the app map. Can I see the details from your app map blade?

[nmakanderhitatchi]

@JacksonWeber Well, not really. It only shows CPU usage but actual network requests won't register - it's just flat even if I put load on it and the services won't show up in the application map. I'll see if I can get you the map too.

[JacksonWeber]

@nmakanderhitatchi Well it makes sense that there are no network requests shown in live metrics if there are none being registered in the transactions blade. When you say "it's flat even if I put load on it" are you referring to the committed memory or the CPU total?

And could I take a look at a code snippet of your express call that should be generating some dependency telemetry?

[nmakanderhitatchi]

@JacksonWeber if I have an endpoint that has app insights v2 and then make requests through insomnia, those requests register in the live metrics, like requests spikes. If I just update to v3, they don't register anymore.

And could I take a look at a code snippet of your express call that should be generating some dependency telemetry? I'm not entirely sure what you mean? We just instantiate App Insights in the way I described it in a previous comment.

[JacksonWeber]

@nmakanderhitatchi I'm just trying to get an idea of what the code you're triggering to generate a request looks like so I can try to repro your issue. A full package.json would also be helpful for this.