Source maps for node typescript + APM

[jeremypress]

Hi, we're working on switching our server to typescript. This causes our APM stack traces to reference the transpiled code and line numbers, instead of the source code. Is there already a strategy to support this? Should we be uploading source maps manually? That looks like a solution just for front end code that was transpiled.

[rochdev]

In order to automatically rewrite every error stack trace with the correct mapping at runtime, you can use source-map-support. This has the advantage of working not only with dd-trace, but also with anything else that captures the stack trace, for example in logs.

[rochdev]

Turns out Node actually supports this out of the box with the --enable-source-maps flag.

[jeremypress]

cool, we'll check out the package. Good to know about the flag, thanks!

[paambaati]

@rochdev Is there guidance on how to get these sourcemaps to Datadog?

For a Node.js app built using TypeScript and compiled using esbuild, we have the .map files generated right next to each .js files. Is this enough, or is there an additional upload step?

[rochdev]

@paambaati It depends if you mean for tracing or for profiling.

For tracing, source maps only make sense in the context of errors. We send whatever stack trace is on the error itself, so it needs to be remapped before the tracer grabs it. You can remap the stack trace of all errors with the --enable-source-maps flag. This also has the benefit that the errors will be mapped properly even outside of tracing, for example in logs.

For profiling, it should be enough to have the source maps present as .map files. cc @Qard to confirm whether this is still the case.

[paambaati]

@rochdev Thanks for the quick response.

We tried the --enable-source-maps flag for error traces, but unfortunately due to https://github.com/nodejs/node/issues/41541, our runtime performance degraded considerably.

Is there another way we can upload sourcemaps for APM error traces?

We do currently have .map.js files emitted next to the source .js files.

[rochdev]

@paambaati Did you also try source-map-support? According to the issue it should be a lot faster and should also work with our tracer.

[ndevvy]

We're trying to do something similar and found that baseline memory usage was very high with the source-map-support library. And we also found --enable-source-maps flag to be untenable per the linked issue above. For now, our solution consists of sending our server errors and source maps to Sentry and will need to cross-reference Sentry issues from APM to get stack traces.

@paambaati -- did you end up getting this working?

[vamsiampolu]

Is it better to generate inline source maps or have .map files with source-map-support for traces?

[paambaati]

@ndevvy It didn't work, and we gave up on it.

[paambaati]

@rochdev It did not work correctly – see https://github.com/evanw/node-source-map-support/issues/122

[rochdev]

We're discussing how to better support this internally, but in the meantime unfortunately the only option is what was mentioned above. If you could share the type of source map you have and the level of transpilation that is done (for example is the code bundled, minified, etc) it would help us with prioritization.

[vamsiampolu]

Node version: LTS 14 transpiled using typescript into a directory that is deployed as a backend service. No bundling or minification is done on our code. We use sourceMap: true to generate the sourcemaps.

[jmoseley]

I am also interested in this. There is some performance concern using --enable-source-maps or source-map-support in production, and ideally we could upload our sourcemaps to Datadog during our CI pipeline, and have the rewrite happen there.

We are running Node 16 and Typescript 4.6.

[alejandromcsd]

On our end, we tried both --enable-source-maps and source-map-support but we weren't able to make it work. Datadog traces still display stack trace from bundle. We are using NestJS in monorepo mode (Node 16 and Typescript 4.3).

[jamesbhobbs]

So many people are reporting this as not working, so why is this closed?

[ndevvy]

We'd really like to switch our backend error monitoring from Sentry to DD and this is the main blocker.

[mrollinsiv]

This is going to quickly become a large issue for us, would love to see some traction on a fix/update.

[muzeke]

Experiencing a similar if not the same issue as well.

Issue description

Source maps in AWS Lambda won't work when i try to use dd-trace in an expressjs project.

Example error:

"Error: There is an error here\n    at /var/task/main.js:2:9517856\n    at Generator.next (<anonymous>)\n    at 
/var/task/main.js:2:9528770\n    at new Promise (<anonymous>)\n    at __awaiter (/var/task/main.js:2:9528331)\n    at 
exports.readExampleData (/var/task/main.js:2:9517469)\n    at Layer.handle_request (/var/task/main.js:2:7053400)\n    at 
next (/var/task/main.js:2:7055283)\n    at Route.dispatch (/var/task/main.js:2:7055319)\n    at Layer.handle_request 
(/var/task/main.js:2:7053400)""

Note : Source maps is working if I don't use dd-trace.

Example error if I'm not using dd-trace:

[dd.trace_id=2878635764044131847 dd.span_id=8063922697396659764] Error message :  webpack://myproject-org-template-microservice/libs/template-microservice/public-api/src/controllers/example/read.ts:19
      throw new Error('There is an error here');
            ^

Error: There is an error here
    at <anonymous> (webpack://myproject-org-template-microservice/libs/template-microservice/public-api/src/controllers/example/read.ts:19:13)
    at Generator.next (<anonymous>)
    at <anonymous> (webpack://myproject-org-template-microservice/node_modules/tslib/tslib.es6.js:121:71)
    at new Promise (<anonymous>)
    at __awaiter (webpack://myproject-org-template-microservice/node_modules/tslib/tslib.es6.js:117:12)
    at fn (webpack://myproject-org-template-microservice/libs/template-microservice/public-api/src/controllers/example/read.ts:9:5)
    at handle_request (webpack://myproject-org-template-microservice/node_modules/express/lib/router/layer.js:95:5)
    at next (webpack://myproject-org-template-microservice/node_modules/express/lib/router/route.js:144:13)
    at fn (webpack://myproject-org-template-microservice/node_modules/express/lib/router/route.js:114:3)
    at Layer.Layer.handle_request (webpack://myproject-org-template-microservice/node_modules/express/lib/router/layer.js:95:5)

• dd-trace version: ^3.19.0
• project is created and build using nx workspace
• Project Details
  • language: typescript
  • framework: expressjs
  • built with nx using @nrwl/node:webpack

[zARODz11z]

Hi @muzeke I'm attempting to reproduce this, may you share specifically how you are using dd-trace? So far my source maps have worked with dd-trace in a lambda function. I am using the lambda layer which uses dd trace under the hood. Are you importing dd trace in your handler for some other use case?

Trying to understand what specific inclusion of dd trace in your app you believe is causing interference with source maps. None the less I have not been able to reproduce your issue.

Any further context is appreciated!

[zARODz11z]

@muzeke here is my stack trace showing src maps with DD Trace SLS Framework DD Lambda Layers:

1. Datadog-Node16-x:91
2. Datadog-Extension:43

I made sure to enable source maps in my webpack config via devtool: 'source-map',

Then in my lambda handler I intentionally throw an error seen below to see if it references the ts file (indicating src maps worked) or the js file (indicating they didn't). Below you can see my stack trace indicates src maps worked. DD Trace works under the hood of the Datadog-Node16 lambda layer which creates downstream express spans, I also have DD Trace working in my lambda handler to create a custom sleep span, none the less I cant reproduce src maps being broken by DD Trace.

ERROR   [dd.trace_id=5364125151088396351 dd.span_id=3217391799624224691] Error handling middleware Error: There is an error here
    at /var/task/dist/webpack:/src/handler.ts:31:15
    at /var/task/node_modules/dd-trace/packages/datadog-instrumentations/src/router.js:50:25
    at /var/task/node_modules/dd-trace/packages/datadog-shimmer/src/shimmer.js:26:21
    at Layer.handle [as handle_request] (/var/task/node_modules/express/lib/router/layer.js:95:5)
    at next (/var/task/node_modules/express/lib/router/route.js:144:13)
    at Route.dispatch (/var/task/node_modules/express/lib/router/route.js:114:3)
    at /var/task/node_modules/dd-trace/packages/datadog-instrumentations/src/router.js:50:25
    at bound dispatch (/var/task/node_modules/dd-trace/packages/datadog-shimmer/src/shimmer.js:26:21)
    at Layer.handle [as handle_request] (/var/task/node_modules/express/lib/router/layer.js:95:5)
    at /var/task/node_modules/express/lib/router/index.js:284:15

[zeke-edamama-ph]

hi @zARODz11z -

When using the datadog extension with lambda, i'm getting source map without any issues, i'm also able to see the traces etc in my datadog, so no issue on that. (If that uses dd-trace under the hood with that setup then i guess for me it works as well.

The error i have above where the source-map isn't showing is when I use dd-trace package in my backend application (express nodejs).

import * as ddtrace from 'dd-trace';

const tracer = ddtrace.tracer;
tracer.init();

// on some endpoints do some tracing stuff 

So if i don't use dd-trace library in my backend service, but have the lambda extension layer , source map is working. But if I use dd-trace to do custom tracing inside my backend service, then the source maps doesn't seem to work.

[zARODz11z]

Hi @zeke-edamama-ph thank you for the response! We are still digging into this. May you give us some more context? Which node version are you using i.e. 16 or 18 Are you using webpack, esbuild ... ?

[mrollinsiv]

@zARODz11z here are some updated details

Serverless Framework Node.js 16.x Datadog-Node16-x: v90 Datadog-Extension-ARM: v41 NODE_OPTIONS: --enable-source-maps serverless-esbuild:

  esbuild:
    packager: yarn
    minify: true
    sourcemap: true

Downloaded the code from AWS and confirmed source map is in place

Logs:

Locally if I run serverless-offline the maps are used and error logs reference the correct files. When I add dd-trace manually and run serverless-offline the sourcemaps no longer work.

[zARODz11z]

In our testing we found using Node 16 to be a possible issue, may you try upgrading to node 18?

[tmadej]

Any update? Anyone have a working solution?

[carlreiser2]

I also have this issue using nodejs 18.x, dd-trace@4.20.0 is installed and implemented. Using source-map-support instead solved my issue

[jonathanmv]

Same for me. I run my app as shown below.

node --require source-map-support/register-hook-require --require dd-trace/init ./dist/src/index.js

Additionally, I made the source maps inline in the tsconfig.json:

    "sourceMap": false,
    "inlineSources": true,
    "inlineSourceMap": true,