DataDog / dd-trace-js

JavaScript APM Tracer
https://docs.datadoghq.com/tracing/
Other
646 stars 306 forks source link

Memory leak v3.5.0 #2448

Closed theo-mesnil closed 1 year ago

theo-mesnil commented 2 years ago

Expected behaviour No memory leak on server side

Actual behaviour We have done a load test with vegeta to check if we got a memory leak and we got this results: Capture d’écran 2022-10-17 à 16 23 00

Steps to reproduce

tracer.init({ service: 'xxxx', profiling: true, runtimeMetrics: true })

export default tracer


- import tracer before all our server code

**Environment**

* **Node.js version:** 16.13.1
* **Tracer version:** 3.5.0
rochdev commented 2 years ago

Did it result in an out-of-memory exception? Also, did the problem appear after a dd-trace upgrade or did you just install the library?

theo-mesnil commented 2 years ago

Hi @rochdev :)

We did a load test before and after dd-trace on our code :

Before dd-trace, result of load testing: Capture d’écran 2022-10-11 à 15 48 06

After dd-trace, result of load testing: Capture d’écran 2022-10-17 à 16 23 00

Qard commented 2 years ago

Looks like you're using profiling. Can you share what your heap profiler data looks like during that load test?

rochdev commented 1 year ago

@theo-mesnil Did you try disabling the profiler? Since you have runtime metrics enabled as well, I'd try to disable that too so that we can narrow down the issue to tracing. Also, we've seen reports of upgrading Node fixing memory leak issues, so might be worth a shot to try upgrading to the latest 16.x. And lastly, did the issue start after upgrading dd-trace or was it a fresh install?

theo-mesnil commented 1 year ago

@Qard @rochdev i'll keep you updated tomorrow after new load testing :)

theo-mesnil commented 1 year ago

We upgraded nodejs for our tests to v16.18.0.

With profiling we got a memory leak who take more time to get the crash: Capture d’écran 2022-11-02 à 17 07 06

When we removed the profiling we got the same result: Capture d’écran 2022-11-02 à 18 20 37

rochdev commented 1 year ago

@theo-mesnil What about runtime metrics and the last version of dd-trace that was not causing a leak?

theo-mesnil commented 1 year ago

@rochdev it's a fresh install for dd-trace, we got the last version 3.7.1 (same with 3.7.0) I removed metrics and profiling and the result:

Capture d’écran 2022-11-03 à 10 27 08

rochdev commented 1 year ago

I would recommend opening a support issue so that we can collect more information about your environment and look into your account.

theo-mesnil commented 1 year ago

Thanks @rochdev i will open a support issue :)

tlhunter commented 1 year ago

@theo-mesnil were you able to open a support issue? If so I'll close this GH issue out.

theo-mesnil commented 1 year ago

Hi @tlhunter, yes thanks i need to make some test with them :)