search

microsoft / ApplicationInsights-Java

Application Insights for Java
http://aka.ms/application-insights
Other
291 stars 196 forks source link

APPLICATIONINSIGHTS_CONNECTION_STRING does not work in 3.5.x, works in 3.4.x #3707

Open mercer opened 1 month ago

mercer commented 1 month ago

Expected behavior

APPLICATIONINSIGHTS_CONNECTION_STRING works in 3.5.x

Actual behavior

APPLICATIONINSIGHTS_CONNECTION_STRING does not work in 3.5.x (tried 3.5.0, 3.5.1, 3.5.2)

To Reproduce

  1. add dependency com.microsoft.azure:applicationinsights-runtime-attach:3.4.19
  2. add applicationinsights.json in src/resources, see bellow
  3. provide connection string via APPLICATIONINSIGHTS_CONNECTION_STRING environment variable
  4. attach to agent via ApplicationInsights.attach();
  5. start app
  6. agent connects to appinsights, traces are sent to apinsights; we had this setup for more than 1 year
  7. upgrade to com.microsoft.azure:applicationinsights-runtime-attach:3.5.2
  8. start app (no other changes, no changes to applicationinsights.json)
  9. agent does not connect, see error bellow in app console
*************************
Application Insights Java Agent 3.5.2 startup failed (PID 46492)
*************************

Description:
Error loading connection string from a file ("unknown-from-application-insights-json").
Please use this format instead:
{ "connectionString": "${file:connection-string-file.txt}" }

Action:
Learn more about configuration options here: https://go.microsoft.com/fwlink/?linkid=2153358

System information

Please provide the following information:

Logs

2023-05-30 17:02:01.940+03:00 ERROR c.m.applicationinsights.agent - Application Insights Java Agent 3.4.13 startup failed (PID 20200)
java.lang.IllegalArgumentException: Missing 'InstrumentationKey'
    at com.azure.monitor.opentelemetry.exporter.implementation.configuration.ConnectionStringBuilder.mapToConnectionConfiguration(ConnectionStringBuilder.java:85)
    at com.azure.monitor.opentelemetry.exporter.implementation.configuration.ConnectionStringBuilder.setConnectionString(ConnectionStringBuilder.java:39)
    at com.azure.monitor.opentelemetry.exporter.implementation.configuration.ConnectionString.lambda$parse$0(ConnectionString.java:37)
    at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)

{
  "connectionString": "unknown-from-application-insights-json",
  "role": {
    "name": "app-role"
  },
  "sampling": {
    "percentage": 100
  },
  "instrumentation": {
    "logging": {
      "level": "INFO"
    }
  },
  "preview": {
    "processors": [
      {
        "type": "attribute",
        "actions": [
          {
            "key": "http.url",
            "pattern": "^(.*)(?<pii>[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,4})(.*)",
            "action": "extract"
          }
        ]
      },
      {
        "type": "attribute",
        "include": {
          "matchType": "strict",
          "attributes": [
            {
              "key": "pii"
            }
          ]
        },
        "actions": [
          {
            "key": "http.url",
            "action": "hash"
          },
          {
            "key": "pii",
            "action": "delete"
          }
        ]
      }
    ],
    "sampling": {
      "overrides": [
        {
          "telemetryType": "request",
          "attributes": [
            {
              "key": "http.url",
              "value": ".*/actuator/health.*",
              "matchType": "regexp"
            }
          ],
          "percentage": 0
        },
        {
          "telemetryType": "request",
          "attributes": [
            {
              "key": "http.url",
              "value": ".*/actuator/dbcheck.*",
              "matchType": "regexp"
            }
          ],
          "percentage": 0
        },
        {
          "telemetryType": "dependency",
          "attributes": [
            {
              "key": "db.system",
              "value": "mssql",
              "matchType": "strict"
            }
          ],
          "percentage": 50
        }
      ]
    }
  }
}
jeanbisutti commented 1 month ago

@mercer I have done some tests and have not noticed any issue with com.microsoft.azure:applicationinsights-runtime-attach:3.5.2.

You have provided this error:

Description:
Error loading connection string from a file ("unknown-from-application-insights-json").
Please use this format instead:
{ "connectionString": "${file:connection-string-file.txt}" }

It could be explained by this JSON config you have provided:

 {
  "connectionString": "unknown-from-application-insights-json",

Could you please check your connection string and retest if something wrong is noticed?

jeanbisutti commented 1 month ago

@mercer The error messages are misleading. Sorry about that. See #3709

mercer commented 1 month ago

I have two issues with this

  1. Isn't this a breaking change that APPLICATIONINSIGHTS_CONNECTION_STRING no longer works with a connection string and now requires a file path? 3.5.x is a minor, not a major semaversion
  2. Documentation has not change, it states that APPLICATIONINSIGHTS_CONNECTION_STRING can be used to provide the connection string at runtime via spring boot's override option

image

jeanbisutti commented 1 month ago

In #3709, only the error message that could be misleading has changed. It does not affect the connection string features.

mercer commented 1 month ago

Then I'd expect my setup to work in 3.5.x just as in 3.4.x, but it doesn't. If you need more details in this ticket, let me know 🙏

jeanbisutti commented 1 month ago

Could you enable the self-diagnostics at the TRACE level (https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config#self-diagnostics) and send the applicationinsights.log file to jeanbisutti@microsoft.com?

trask commented 1 month ago

@mercer any luck here? we'd love to understand if something broke in 3.5.x

mercer commented 3 weeks ago

build.gradel that works

plugins {
    id "io.franzbecker.gradle-lombok" version "5.0.0"
    id "org.springframework.boot" version "2.7.18"
    id "org.sonarqube" version "4.0.0.2929"
    id "org.owasp.dependencycheck" version "8.1.2"
    id "com.github.johnrengelman.shadow" version "8.1.0"
}

repositories {
    mavenCentral()
    maven { url "https://oss.jfrog.org/artifactory/oss-snapshot-local/" }
}
apply plugin: "java"
apply plugin: "jacoco"
apply plugin: "io.spring.dependency-management"

group = "com.redacted"
version = project.getProperty("version")
sourceCompatibility = "17"

jacoco {
    toolVersion = "0.8.8"
    reportsDirectory = file("$buildDir/customJacocoReportDir")
}

jacocoTestReport {
    reports {
        xml.required = true
        csv.required = false
        html.outputLocation = file("${buildDir}/jacocoHtml")
    }
}

test.finalizedBy jacocoTestReport

lombok { // optional: values below are the defaults
    version = "1.18.24"
    sha256 = "d3584bc2db03f059f984fb0a9c119aac1fa0da578a448e69fc3f68b36584c749"
}

dependencyManagement {
    imports {
        mavenBom "org.springframework.boot:spring-boot-dependencies:2.7.18"
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:2021.0.3"
    }
    dependencies {
        // these dependencies are not in spring boot or spring cloud BOMs
        dependency("org.springframework.boot:spring-boot-starter-batch:2.7.18")  //addresses  CVE-2023-20873, CVE-2023-20883, CVE-2023-34034
        dependency("com.microsoft.azure:applicationinsights-runtime-attach:3.4.19")
        dependency("io.opentelemetry:opentelemetry-sdk:1.38.0")
        dependency("com.azure.spring:spring-cloud-azure-starter-keyvault-secrets:4.16.0")
        dependency("com.azure:azure-messaging-servicebus:7.15.2")
        dependency("org.springdoc:springdoc-openapi-webmvc-core:1.7.0")
        dependency("org.springdoc:springdoc-openapi-security:1.7.0")
        dependency("org.springdoc:springdoc-openapi-ui:1.7.0")
        dependency("com.sun.xml.bind:jaxb-core:2.3.0.1")
        dependency("com.sun.xml.bind:jaxb-impl:2.3.3")
        dependency("org.owasp:security-logging-logback:1.1.7")
        dependency("org.apache.commons:commons-text:1.10.0")
        dependency("com.c4-soft.springaddons:spring-security-test-oauth2-addons:1.2.0")
        dependency("org.powermock:powermock-reflect:2.0.9")
        dependency("com.frejo:force-rest-api:0.0.44")
        dependency("net.javacrumbs.shedlock:shedlock-spring:4.34.0")
        dependency("net.javacrumbs.shedlock:shedlock-provider-jdbc-template:4.34.0")
        dependency("com.microsoft.sqlserver:mssql-jdbc_auth:10.2.1.x64")
        dependency("com.vladmihalcea:hibernate-types-52:2.12.1")
        dependency("com.launchdarkly:launchdarkly-java-server-sdk:5.9.0")
        dependency("com.google.guava:guava:33.1.0-jre")  // fix for CVE-2023-2976
        dependency("com.google.code.gson:gson:2.10.1")
        dependency("commons-io:commons-io:2.15.1")

        // these dependencies are in spring boot, but downgraded to fix a compatibility issue
        dependency("com.h2database:h2:1.4.200")

        //these dependencies overwrite the default bom, please review and remove after upgrading spring boot:
        dependency("org.yaml:snakeyaml:2.0")   // fix for CVE-2022-1471
        dependency("com.fasterxml.jackson.core:jackson-core:2.15.0")   // fix for CVE-2022-1471
        dependency("org.liquibase:liquibase-core:4.25.1")  // fix for CVE-2022-1471
        dependency("org.springframework.security:spring-security-core:5.7.12")     // fix for CVE-2016-1000027
        dependency("org.springframework:spring-web:5.3.33") //fix for CVE-2024-22243, CVE-2024-22259, CVE-2016-1000027
        dependency("ch.qos.logback:logback-core:1.2.13")  // fix for CVE-2023-6481
        dependency("ch.qos.logback:logback-classic:1.2.13") // fix CVE-2023-6378
        dependency("io.netty:netty-codec-http:4.1.108.Final") // fix CVE-2024-29025
    }
}

dependencies {
    implementation('org.springframework.boot:spring-boot-starter-validation')
    implementation("com.azure.spring:spring-cloud-azure-starter-keyvault-secrets")
    implementation("com.microsoft.azure:applicationinsights-runtime-attach")
    implementation("io.opentelemetry:opentelemetry-sdk")
    implementation("org.springframework.cloud:spring-cloud-starter-sleuth")
    implementation("org.springframework.boot:spring-boot-starter-web")
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    implementation("org.springframework.boot:spring-boot-starter-data-jpa")
    implementation("org.springframework.boot:spring-boot-starter-hateoas")
    implementation("org.springframework.boot:spring-boot-starter-security")
    implementation("org.springframework.boot:spring-boot-starter-batch")
    implementation("org.springframework.security:spring-security-oauth2-jose")
    implementation("org.springframework.security:spring-security-oauth2-resource-server")
    implementation("org.liquibase:liquibase-core")
    implementation("org.springdoc:springdoc-openapi-webmvc-core")
    implementation("org.springdoc:springdoc-openapi-security")
    implementation("org.springdoc:springdoc-openapi-ui")
    implementation("com.sun.xml.bind:jaxb-impl")
    implementation("com.sun.xml.bind:jaxb-core")
    implementation("com.azure:azure-messaging-servicebus") {
        exclude group: "org.slf4j", module: "slf4j-log4j12"
    }
    implementation("org.owasp:security-logging-logback")
    implementation("com.frejo:force-rest-api")
    implementation("com.fasterxml.jackson.datatype:jackson-datatype-jsr310")
    implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-csv")
    implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-xml")
    implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
    implementation("org.apache.httpcomponents:httpclient")
    implementation("org.apache.commons:commons-text")
    implementation("net.javacrumbs.shedlock:shedlock-spring")
    implementation("net.javacrumbs.shedlock:shedlock-provider-jdbc-template")
    implementation('com.vladmihalcea:hibernate-types-52')
    implementation('org.hibernate:hibernate-core')
    implementation('com.launchdarkly:launchdarkly-java-server-sdk')
    implementation('org.apache.commons:commons-lang3')
    implementation('com.google.guava:guava')
    implementation('com.google.code.gson:gson')
    implementation('commons-io:commons-io')
    implementation("io.netty:netty-codec-http")

    runtimeOnly("com.microsoft.sqlserver:mssql-jdbc")
    if (System.getProperty('env') == 'dev') {
        runtimeOnly("com.h2database:h2")
    } else {
        testImplementation("com.h2database:h2")
    }

    testImplementation("org.junit.jupiter:junit-jupiter-engine")
    testImplementation("org.junit.jupiter:junit-jupiter-params")
    testImplementation("org.mockito:mockito-junit-jupiter")
    testImplementation("org.springframework.boot:spring-boot-starter-web") {
        exclude group: "junit", module: "junit"
    }
    testImplementation("org.springframework.boot:spring-boot-starter-test") {
        exclude group: "junit", module: "junit"
        exclude group: "org.junit.vintage", module: "junit-vintage-engine"
    }
    testImplementation("com.c4-soft.springaddons:spring-security-test-oauth2-addons") {
        exclude group: "junit", module: "junit"
    }
    testImplementation("org.awaitility:awaitility") {
        exclude group: "junit", module: "junit"
    }
    testImplementation("org.powermock:powermock-reflect") {
        exclude group: "junit", module: "junit"
    }
}

dependencyCheck {
    suppressionFile = "$projectDir/owasp-suppression.xml"
}

test {
    useJUnitPlatform()
    maxHeapSize = "4096m"
}

shadowJar {
    zip64 true
}

build.gradle that doesn't work

......
dependencyManagement {
    imports {
        mavenBom "org.springframework.boot:spring-boot-dependencies:2.7.18"
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:2021.0.3"
    }
    dependencies {
        // these dependencies are not in spring boot or spring cloud BOMs
        dependency("org.springframework.boot:spring-boot-starter-batch:2.7.18")  //addresses  CVE-2023-20873, CVE-2023-20883, CVE-2023-34034
        dependency("com.microsoft.azure:applicationinsights-runtime-attach:3.5.3")
.............

applicationinsights.log

App log output when it starts, only tested locally, the only change is the upgrade from 3.4.19 to 3.5.3.

/Library/Java/JavaVirtualMachines/zulu-17.jdk/Contents/Home/bin/java -XX:TieredStopAtLevel=1 -Dspring.profiles.active=dev-personal -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true -Dmanagement.endpoints.jmx.exposure.include=* -javaagent:/Users/emilian/Applications/IntelliJ IDEA Ultimate.app/Contents/lib/idea_rt.jar=54330:/Users/emilian/Applications/IntelliJ IDEA Ultimate.app/Contents/bin -Dfile.encoding=UTF-8 @/private/var/folders/lb/vq9z2h5j1ygcq4mlym46pfp00000gn/T/idea_arg_file1879499461 com.redacted.RedactedApiSpringBoot
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

*************************
Application Insights Java Agent 3.5.3 startup failed (PID 48831)
*************************

Description:
Your connection string seems to have a wrong format: "unknown-from-application-insights-json").
If you want to load the connection string from a file, please use this format:
{ "connectionString": "${file:connection-string-file.txt}" }

Action:
Learn more about configuration options here: https://go.microsoft.com/fwlink/?linkid=2153358
mercer commented 3 weeks ago

With 3.4.19 I see this applicationinsights.log

2024-06-05 16:27:59.395+03:00 INFO  c.m.applicationinsights.agent - Application Insights Java Agent 3.4.19 started successfully (PID 49245, JVM running for 1.627 s)
2024-06-05 16:27:59.399+03:00 INFO  c.m.applicationinsights.agent - Java version: 17.0.7, vendor: Azul Systems, Inc., home: /Library/Java/JavaVirtualMachines/zulu-17.jdk/Contents/Home
jeanbisutti commented 5 days ago

HI @mercer, Sorry for the delay. I was off. I had tried with a Petclinic application and Application Insights Runtime Attach 3.5.2 without observing any issue. To be able to help you, could you please point to a Github repository reproduding the issue or send by email the Application Insights logs at the TRACE level as previously asked?

Cloud5000 commented 4 days ago

I can confirm what @mercer mentioned. APPLICATIONINSIGHTS_CONNECTION_STRING works with 3.4.19 but not with 3.5.x.

jeanbisutti commented 4 days ago

Hi @mercer and @Cloud5000

I have retested the Petclinic application with the Runtime Attachment feature and with the APPLICATIONINSIGHTS_CONNECTION_STRING environment variable. I can see the telemetry data on the Azure portal with the 3.5.2 and 3.5.3 versions.

The only way to reproduce the same error message seen by @mercer is to use an invalid connection string in the applicationinsights.json file:

*************************
Application Insights Java Agent 3.5.3 startup failed (PID 4336)
*************************

Description:
Your connection string seems to have a wrong format: "unknown-from-application-insights-json").
If you want to load the connection string from a file, please use this format:
{ "connectionString": "${file:connection-string-file.txt}" }

Could you please remove the following connection string from the applicationinsights.json file:

{

  "connectionString": "unknown-from-application-insights-json"
}
Cloud5000 commented 4 days ago

Thank you @jeanbisutti, removing the connectionString property helped.

mercer commented 4 days ago

Ok, but this is a regression. As this is semver, a minor version change should not introduce a breaking change.