search

elastic / ecs-logging-nodejs

https://www.elastic.co/guide/en/ecs-logging/nodejs/master/intro.html
Apache License 2.0
68 stars 39 forks source link

ecs-pino-format does not integrate with elastic-apm-node when it is wrapped inside a library #130

Open sriram-kailasam opened 2 years ago

sriram-kailasam commented 2 years ago

Hi. We have an internal NPM package for logging, which wraps Pino and ecs-pino-format to format logs. We are using this library in multiple services. We also use elastic-apm-node in these services.

The problem is that the automatic integration of elastic-apm-node doesn't work when ecs-pino-format is wrapped inside this library, but it works when I directly import Pino and log from the service. trace.id and event.dataset fields are not added to the logs that come from the library.

What I'm using:

Typescript in both the service and the library.

"@elastic/ecs-pino-format": "^1.1.2" "elastic-apm-node": "^3.9.0"

Example to explain what I mean:

I created 2 folders, one to simulate the service and the other to simulate the library. I npm linked the library with the service.

pino-format-issue-service:

image

pino-format-issue-lib:

image

Log output:

{"log.level":"info","@timestamp":"2022-10-25T09:16:28.411Z","process":{"pid":71004},"host":{"hostname":"<redacted>"},"ecs":{"version":"1.6.0"},"service":{"name":"test-service"},"event":{"dataset":"test-service.log"},"message":"info from pino"}
{"log.level":"info","@timestamp":"2022-10-25T09:16:28.412Z","process":{"pid":71004},"host":{"hostname":"<redacted>"},"ecs":{"version":"1.6.0"},"message":"info from lib"}

As you can see, event.dataset field is added to the first log (directly logging from pino), but it's not there in the second log.

What I've tried:

  1. Importing elastic-apm-node/start in both the service and the library. This adds trace.id to the logs from library as well, but the ids from the service and library do not match as the agent is imported twice. This defeats the purpose as I will not be able to jump from APM to the log.

  2. Importing the agent only inside the library, and removing it from the service. I would be fine with doing this, but it seems that incoming and outgoing requests are not traced automatically when I do this, so there is no trace on APM.

  3. Sending the request and response to the library using a middleware, and adding trace.id manually to the logs. This honestly seems too complicated, and even led to a memory leak in the code that maintains request and response using async_hooks.

Is there a simpler way to achieve this that I'm missing? I really want to integrate APM and Logs for better debugging, and I also don't want to remove this library and drop to using bare Pino.

Thanks.

ThomasBouasli commented 6 months ago

I have the same problem, apparently what happens is that the code first looks for the serviceName, since you did not pass it, it tries to get it from APM, but it is lazy loaded, so it is undefined when that check happens, leaving you without a serviceName. That does not happen when you dont use the wrapper library probably because the apm is already loaded in that context, i'm not really sure. #79 introduced this bug.