Feature Request: Application Insights / Distributed tracing support

ezYakaEagle442 commented 2 years ago

Is your feature request related to a problem? Please describe.
Azure Container Apps does not provide a Distributed tracing solution, like Java In-Process Agent which is supported in Azure Spring Apps

Describe the solution you'd like.
When designing micro-services and connecting multiple apps, with or WITHOUT Dapr, SRE/Ops need a Distributed tracing solution integrated in the Azure portal

Describe alternatives you've considered.
None

Additional context.
N/A

BigMorty commented 2 years ago

While Azure Container Apps does not have a no-code way to integrate App Insights today, you can still use the App Insights SDK and get the same benefits.

bengimblett commented 2 years ago

@BigMorty I am also struggling here

I have a simple two microservices (dotnet core minimal api, latest) each with AI SDK latest and they communicate via DAPR and an SB Queue binding

I set the following when deploying the ACA Env daprAIInstrumentationKey

I therefore have 3 x AI resources - one for DAPR, one for each microservice

So whilst i do see telemetry none of it is correlated each item has its own operation id (which is incorrect)

I had expected end to end correlation (an app map of each component connected by DAPR)

Looking at the DAPR docs the config for AI is via OTEL but this appears abstracted away for ACA

Can you help ? Is this supposed to work in the way i assumed it would?

kendallroden commented 2 years ago

You should see an app map in the resource you provided the instrumentation key for- can you share any additional information on your config? How did you deploy this solution, etc.?

ezYakaEagle442 commented 2 years ago

@kendallroden I have understood now, it should work : in my case I do not use Dapr, I have provisionned AppInsights with Bicep , then I use its ConnectionString in my main.bicep where a Module create the ACA Apps inside which I use the ConnectionString as env :

resource AdminServerContainerApp 'Microsoft.App/containerApps@2022-03-01' = {
  name: adminServerContainerAppName
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    managedEnvironmentId: corpManagedEnvironment.id 
    configuration: {
      activeRevisionsMode: 'Multiple'
      ingress: {
        allowInsecure: true
        external: true
        targetPort: 9090
        traffic: [
          {
            latestRevision: true
            revisionName: revisionName
            weight: 100
          }
        ]
        transport: 'auto'
      }
      registries: [
        {
          passwordSecretRef: 'registrypassword'
          server: ghaSettingsCfgRegistryUrl
          username: ghaSettingsCfgRegistryUserName
        }
      ]
      secrets: [
        {
          name: 'registrypassword'
          value: ghaSettingsCfgRegistryPassword
        }
        {
          name: 'appinscon'
          value: appInsightsInstrumentationKey
        }
    template: {
      containers: [
        {
          command: [
            'ENTRYPOINT ["java", "-javaagent:${applicationInsightsAgentJarFilePath}", "org.springframework.boot.loader.JarLauncher", "--server.port=8080", "--spring.profiles.active=docker,mysql"]'
          ]
          env: [
            {
              name: 'SPRING_PROFILES_ACTIVE'
              value: 'docker,mysql'
            }
            {
              // https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent#set-the-application-insights-connection-string
              name: 'APPLICATIONINSIGHTS_CONNECTION_STRING'
              secretRef: 'appinscon'
            }

bengimblett commented 2 years ago

You should see an app map in the resource you provided the instrumentation key for- can you share any additional information on your config? How did you deploy this solution, etc.?

Thanks - iit was bicep + GH Actions. The services are dotnet latest and the invocation was via DAPR (abstracting ASB Queue) I will redeploy and try again, I'll DM you if i see same

bengimblett commented 2 years ago

@kendallroden

In a simple test I have two dotnet web api in an ACA env. Each web app has its own app insights (via sdk and connection string env var config). The ACA Env is also linked to it's own AI resource.

Overview

Resources components

Bicep for env

resource environment 'Microsoft.App/managedEnvironments@2022-03-01' = { name: environmentName location: location properties: { daprAIInstrumentationKey: reference(appInsights.id, '2020-02-02').InstrumentationKey appLogsConfiguration: { destination: 'log-analytics' logAnalyticsConfiguration: { customerId: reference(workspaceId).customerId sharedKey: logAnalyticsWorkspace.listKeys().primarySharedKey } }

What I'm seeing is telemetry in all 3 but not linked. The operations id is unique. Therefore the app map and end-to-end view isnt working. The gap seems to be with the env/dapr.

This is the app map from the dapr ai resource - the E2E view isnt working the map has some of the info, but the target api isnt linked appmap-from-env-ai

The screen snip was a good 15mins after the services were invoked (giving enough time for data to filter through)

Am i doing anything wrong here? What should be happening is the http call from the ACA App to DAPR should have the traceparent header and then DAPR should do the rest. I cant easily repro this outside of ACA, running locally via Docker compose for example, as i'd need to know more exactly what you did on the AKS hosting side (presumably using otel) and I dont :)

bengimblett commented 2 years ago

This didnt work in a slightly more complex test with ASB via DAPR (queue) either.

The appmap should show the services linked and the E2E view should be showing the linked up view too (again this should be the same at least for the two service AI resources)

is the wc3 traceparent getting dropped if the dapr sidecar is using grpc (which i think is the default) ?

bengimblett commented 2 years ago

OK I waited four hours and came back to it, the test for api to api via dapr and service invocation - in case there was some delay, so ....

For each API App insights resource i see the same E2E view for a given request telemetry (this is good and expected but normally this only takes max 5mins to show) . however, the app map does NOT show the full view in either C-App AI resource

If i move to the third AI resource (the one for dapr against the environment) after these several hours end to end view is also correct (which makes sense) AND the app map does now show correct.

So the issue is (1) App Map for some reason only shows on the one resource (the App insights against the env+dapr) (2) There is a LONG delay for the Env/AI telemetry to be processed - and without this the E2E for each C-App/api and the Map wont work

ezYakaEagle442 commented 1 year ago

@BigMorty I deploy Java Apps to ACA, I have set correctly the AppInsights config files applicationinsights.json with 1 unique name for each App. Ex at https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/blob/main/spring-petclinic-customers-service/src/main/resources/applicationinsights.json

AppInsight Agent for Java in ACA shows only 1 green circle in Application Map. Could you please tell me what is wrong in my config ?

The Application Map looks like this :

Whereas it should looks like this :

The Container command is set in Bicep with:

      secrets: [
        // [https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent#set-the-application-insights-connection-string](https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent#set-the-application-insights-connection-string)
        {
          name: 'appinscon'
          value: appInsights.properties.ConnectionString
        }

    template: {
      containers: [
        { 
          command: [
            'java', '-javaagent:${applicationInsightsAgentJarFilePath}', 'org.springframework.boot.loader.JarLauncher', '--server.port=8080', '-Xms512m -Xmx1024m'
          ]
          env: [
            {
              name: 'SPRING_PROFILES_ACTIVE'
              value: 'docker,mysql'
            }
            {
            // [https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent#set-the-application-insights-connection-string](https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent#set-the-application-insights-connection-string)
        {
              name: 'APPLICATIONINSIGHTS_CONNECTION_STRING'
              secretRef: 'appinscon'
            }

ezYakaEagle442 commented 1 year ago

@BigMorty @sanchitmehta is it because I set this config https://github.com/ezYakaEagle442/aca-java-petclinic-mic-srv/blob/main/iac/bicep/modules/aca/acaPublicEnv.bicep#L22 with destination:log-analytics ?:

@allowed([
  'log-analytics'
  'azure-monitor'
])
@description('Cluster configuration which enables the log daemon to export app logs to a destination. Currently only "log-analytics" is supported https://learn.microsoft.com/en-us/azure/templates/microsoft.app/managedenvironments?pivots=deployment-language-bicep#managedenvironmentproperties')
param logDestination string = 'log-analytics' // 'log-analytics' or 'azure-monitor' 

resource logAnalyticsWorkspace  'Microsoft.OperationalInsights/workspaces@2021-12-01-preview' existing =  {
  name: logAnalyticsWorkspaceName
}

resource appInsights 'Microsoft.Insights/components@2020-02-02-preview' existing = {
  name: appInsightsName
}

/*
ACA does not yet support diagnostic settings
container apps do no support currently diagnostic settings. Integration happen trough property on the app environment resource currently.
https://github.com/microsoft/azure-container-apps/issues/382
https://docs.microsoft.com/en-us/azure/templates/microsoft.insights/diagnosticsettings?tabs=bicep
*/
resource corpManagedEnvironment 'Microsoft.App/managedEnvironments@2022-06-01-preview' = {
  name: azureContainerAppEnvName
  location: location
  properties: {
    appLogsConfiguration: {
      destination: logDestination // azure-monitor
      logAnalyticsConfiguration: {
        customerId: logAnalyticsWorkspace.properties.customerId
        sharedKey: logAnalyticsWorkspace.listKeys().primarySharedKey
      }
    }
    zoneRedundant: zoneRedundant
    daprAIInstrumentationKey: appInsights.properties.InstrumentationKey
    daprAIConnectionString: appInsights.properties.ConnectionString
  }
}

output corpManagedEnvironmentId string = corpManagedEnvironment.id 
output corpManagedEnvironmentDefaultDomain string = corpManagedEnvironment.properties.defaultDomain
output corpManagedEnvironmentStaticIp string = corpManagedEnvironment.properties.staticIp

Would this fix the issue ?:

param logDestination string = 'azure-monitor' 

resource appInsightsDiagnosticSettings 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: 'dgs-${appName}-send-${azureContainerAppEnvName}-logs-and-metrics-to-log-analytics'
  scope: corpManagedEnvironment
  properties: {
    logAnalyticsDestinationType: 'AzureDiagnostics'
    workspaceId: logAnalyticsWorkspace.id
    logs: [
      {
        category: 'ContainerAppConsoleLogs'
        enabled: true
      }
      {
        category: 'ContainerAppSystemLogs'
        enabled: true
      }      
    ]
    metrics: [
      {
        category: 'AllMetrics'
        enabled: true
        retentionPolicy: {
          days: 7
          enabled: true
        }
      }
    ]
  }
}

ezYakaEagle442 commented 1 year ago

@BigMorty @sanchitmehta

As stated in the docs , by default, Application Insights Java 3.x expects the configuration file to be named applicationinsights.json and to be located in the same directory as applicationinsights-agent-3.x.x.jar

You can specify your own configuration file path by using one of these two options:

APPLICATIONINSIGHTS_CONFIGURATION_FILE environment variable
applicationinsights.configuration.file Java system property

In my configuration:

applicationinsights-agent-3.4.1.jar is located in the container at /tmp/app/applicationinsights-agent-3.4.1.jar
In the source code, the config files are located in each micro-service at src/main/resources/applicationinsights.json
in the ACA containers the applicationinsights.json is located at BOOT-INF/classes/applicationinsights.json because in the Dockerfile I use Java layertools to extract all the files from the jar.

setting this Env. var APPLICATIONINSIGHTS_CONFIGURATION_FILE=BOOT-INF/classes/applicationinsights.json fixed the issue, I then see each service in a green circle in the AppMap EXCEPT Key-Vault whereas each service get the DB connection string secret from KV

SophCarp commented 1 year ago

There's been a lot of discussion on this thread. Has the issue been resolved? @ezYakaEagle442

ghost commented 1 year ago

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

edeandrea commented 1 year ago

I see this issue was closed but I don't see a solution mentioned in the thread. Was there ever a distributed tracing solution added?

microsoft / azure-container-apps

Feature Request: Application Insights / Distributed tracing support #290