dd-lib-java-init overrides the value of env JAVA_TOOL_OPTIONS

[gluckzhang]

Describe what happened: When I use the recommended approach to add the instrumentation library for my Java services. I find that dd-lib-java-init overrides the value of JAVA_TOOL_OPTIONS in order to add the -javaagent option to the JVM. The previously configured values no longer exist.

Describe what you expected: Instead of something like JAVA_TOOL_OPTIONS="-javaagent:/datadog-lib/dd-java-agent.jar", I hope to use JAVA_TOOL_OPTIONS="$JAVA_TOOL_OPTIONS -javaagent:/datadog-lib/dd-java-agent.jar". In this way, we are free to add additional options to the JVM using this standard env var JAVA_TOOL_OPTIONS.

Steps to reproduce the issue:

• Add the following label and annotation to the service's deployment

  template:
  metadata:
  labels:
      admission.datadoghq.com/enabled: "true"
  annotations:
      admission.datadoghq.com/java-lib.version: "v1.8.0"

• Check the deployment and you will see:

  Containers:
  server:
  ....
  Environment:
    ....
    JAVA_TOOL_OPTIONS:   -javaagent:/datadog-lib/dd-java-agent.jar
    ....

  In this service's image, we have instructions to set up JAVA_TOOL_OPTIONS, which is overridden by the init container.

Additional environment details (Operating System, Cloud provider, etc): dd-lib-java-init v1.8.0, in AWS EKS

[clamoriniere]

Hi @gluckzhang

Thanks for opening this issue. The problem that you are describing is not directly linked to the helm chart that you use to deployment the agent, but more about how the admission controller in the cluster-agent inject or update the "JAVA_TOOL_OPTIONS" environment variable. So this issue should have been create on the http://github.com/datadog/datadog-agent repository.

We have investigated your suggestion to set

env:
- name:  JAVA_TOOL_OPTIONS
  value: "$JAVA_TOOL_OPTIONS -javaagent:/datadog-lib/dd-java-agent.jar"

and

env:
- name:  JAVA_TOOL_OPTIONS
  value: "$(JAVA_TOOL_OPTIONS) -javaagent:/datadog-lib/dd-java-agent.jar"

But unfortunately it doesn't work, because the kubelet is not able to access and replace environment variable defined inside the dockerfile.

As a workaround we can suggested to define your initial JAVA_TOOL_OPTIONS configuration on the pod's container envvar. In that case the "admission controller" will append properly the datadog jar to the envvar that you have configured.

Please let us know if this workaround works for you 🙇

Also, if you need more support from us, please contact please contact Datadog support

[gluckzhang]

Hi @clamoriniere , thank you very much for the reply and the investigation! I understand it now. We will try the workaround and keep you posted. I think it will work well if we configure JAVA_TOOL_OPTIONS in a deployment instead of in a Dockerfile :)

[ewerkema]

We ran into the same problem and can verify that the solution by including JAVA_TOOL_OPTIONS from the deployment environment variables works:

spec:
  containers:
    - name: ...
      env:
        - name: JAVA_TOOL_OPTIONS
          value: YOUR_CUSTOM_OPTIONS

It then successfully merges the datadog options with our custom deployment options.