search

CiscoDevNet / appdynamics-charts

Helm charts for AppDynamics
https://appdynamics.github.io/appdynamics-charts/
Apache License 2.0
21 stars 38 forks source link

AppDynamics cluster agent auto-instrumentation breaks other deployed Helm releases #12

Closed rootd00d closed 3 years ago

rootd00d commented 3 years ago

The AppDynamics cluster agent appears to work by running an init container that copies the AppDynamics language-specific agent and instrumentation shim into the application container. It does so by creating an emptyDir volume named "appd-agent-repo-nodejs" (in our case, anyways) that is mounted by the init container, and the application container. This appears to be accomplished by modifying the deployment spec to define the volume and have the application container mount the volume.

A problem arises when attempting to deploy the service again, as it will generate the following error -- 

Error: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: cannot patch "foo" with kind Deployment: Deployment.apps "foo" is invalid: spec.template.spec.initContainers[0].volumeMounts[0].name: Not found: "appd-agent-repo-nodejs": release foo failed: timed out waiting for the condition

This is because the deployment spec has been updated and the application Helm chart does not match anymore. One workaround was to try adding the "appd-agent-repo-nodejs" emptyDir volume manually to each service Helm chart, which seemed to work, at least for deploying the chart. However, when the containers would actually try to start the AppDynamics cluster agent seemed to skip actually performing the instrumentation, and the services would fail to start with this error -- 

internal/modules/cjs/loader.js:796
    throw err;
    ^

Error: Cannot find module '/opt/appdynamics-nodejs/shim.js'
Require stack:
- internal/preload
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:793:17)
    at Function.Module._load (internal/modules/cjs/loader.js:686:27)
    at Module.require (internal/modules/cjs/loader.js:848:19)
    at Module._preloadModules (internal/modules/cjs/loader.js:1133:12)
    at loadPreloadModules (internal/bootstrap/pre_execution.js:443:5)
    at prepareMainThreadExecution (internal/bootstrap/pre_execution.js:62:3)
    at internal/main/run_main_module.js:7:1 {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ 'internal/preload' ]
rootd00d commented 3 years ago

I was 90% through writing this issue when I decided to ensure that our Helm versions were fully up to date. After upgrading to Helm 3.4.2, this doesn't appear to be a problem anymore. The issue was likely something Helm-specific, and this kind of init container rewriting of the deployment spec appears to be supported now.

I decided to finish filing the issue in case someone else out there hits this. When looking for a solution I wasn't able to find anything to steer me in the right direction, so hopefully this might help someone out in the future.