APM Not working, data not show up in Kibana

[fathuraw]

Describe the bug

APM Data is not showing up / sent to APM Server in Elastic Cloud when running inside Kubernetes Container.

To Reproduce Steps to reproduce the behavior:

1. Elastic Cloud Deployment version: v8.0.0

2. Node Application Config node: v16.14.0 express: v4.17.1 typescript: v4.2.3 elastic-apm-node: v3.29.0 OS: Pods using node:16-slim as Base Image, deployed on GKE

How are you starting the agent? (please tick one of the boxes) We run the docker from CMD bash ./entrypoint.sh

the sh script will load the env and run node index.js that start with code below:

import apm from 'elastic-apm-node'
apm.start({
  // Override service name from package.json
  // Allowed characters: a-z, A-Z, 0-9, -, _, and space
  serviceName: 'service-name',
  environment: 'production',
  active: true,
  secretToken: '<hardcoded>',
  serverUrl: '<hardcoded>'
})

import './App'

Additional context

• When running with node dist/index.js or ELASTIC_APM_ACTIVE=true ELASTIC_APM_SECRET_TOKEN=<token> ELASTIC_APM_SERVER_URL=<apmurl> pm2 start --name implikasi dist/index.js, the apm agent run fine, and data is shown in Kibana Dashbard But when I run inside Docker/Kubernetes, there's no metrics sent.
• We hardcoded the APM config, still not working

Expected behavior APM Data show up in Kibana Observability Dashboard

Is there any step that I missed? I'm confused what is the root cause of this.

[astorm]

Hello @fathuraw, nice to meet you, and thanks for the detailed information about your deployment architecture. That information makes it much easier to get started with debugging your connection issue.

To start -- when folks are using compiled javascript code we generally advise that they preload the agent via node -r elastic-apm-node/start and use env. vars for configuration (since start.js will start the agent). This ensures the agenet is the first thing loaded in their dist source. However, based on what you've said I don't think this is your issue, since the application works when you invoke it directly. I mention this mainly for future readers, and in case your build produces a different dist inside your docker container.

With that said, the next most likely suspect is your container setup. My guess would be that your individual container networking isn't setup to allow for your containers to talk to the APM Server endpoints. The fastest way to determine if this is case will be to deploy with logging set to the trace level. If there's networking issue they'll show up in your K8s/container logs. If there's no networking issues, the trace logging may point to the issue.

Also -- the GitHub issues here are generally for discussions around features and bugs with the agent. We're always happy to lend a hand with setup questions when we have a moment, but you may get a faster/better answer by posting this to our discuss forums or engaging with support directly.

Does that make sense/get you headed in the right direction?

[fathuraw]

Hello @fathuraw, nice to meet you, and thanks for the detailed information about your deployment architecture. That information makes it much easier to get started with debugging your connection issue.

To start -- when folks are using compiled javascript code we generally advise that they preload the agent via node -r elastic-apm-node/start and use env. vars for configuration (since start.js will start the agent). This ensures the agenet is the first thing loaded in their dist source. However, based on what you've said I don't think this is your issue, since the application works when you invoke it directly. I mention this mainly for future readers, and in case your build produces a different dist inside your docker container.

With that said, the next most likely suspect is your container setup. My guess would be that your individual container networking isn't setup to allow for your containers to talk to the APM Server endpoints. The fastest way to determine if this is case will be to deploy with logging set to the trace level. If there's networking issue they'll show up in your K8s/container logs. If there's no networking issues, the trace logging may point to the issue.

Also -- the GitHub issues here are generally for discussions around features and bugs with the agent. We're always happy to lend a hand with setup questions when we have a moment, but you may get a faster/better answer by posting this to our discuss forums or engaging with support directly.

Does that make sense/get you headed in the right direction?

Hi astorm, sorry for the very late reply. Moving the start function to node -r elastic-apm-node/start is working for typescript project. Thank you!