tracing & profile feature can not working well

[luyi0521]

Expected behaviour

collect tracing & profiling data

Actual behaviour

1. have tracing data but no profiling data try upgrade dd-trace profiling data collected.(Not aware we lost part of tracing data) Tracing page on portal haven't service-graphql tracing records.
2. try update tracer to fix tracing data collect issue
3. rollback to 0.x version tracer to fix tracing data collecting. But profiling data still not have

Steps to reproduce

upgrade dd-trace from 0.33.1 to 2.10.0 upgrade from 2.10.0 to 2.12.2 downgrade from 2.12.2 to 0.36.6

Environment

• Operation system: Alpine (Pod on Kubernetes)
• Node.js version: 16.15.0
• Tracer version: 2.12.2,2.10.0
• Agent version: 7.32.4
• Relevant library versions: graphql@14.7.0

dependencies in package.json "@google-cloud/dialogflow": "^3.3.0", "@google-cloud/pubsub": "^2.11.0", "@google-cloud/storage": "^5.8.3", "@google-cloud/trace-agent": "^4.2.5", "@walletpass/pass-js": "^6.9.1", "apn": "^2.2.0", "apollo-errors": "^1.9.0", "apollo-link": "^1.2.14", "apollo-link-error": "^1.1.13", "apollo-link-http": "^1.5.17", "apollo-server-express": "^2.4.12", "apollo-server-plugin-base": "^0.9.1", "axios": "^0.27.2", "chance": "^1.0.18", "dataloader": "^2.0.0", "dd-trace": "0.36.6", "express": "^4.16.3", "express-csp-header": "^4.1.0", "firebase-admin": "^9.2.0", "generic-pool": "^3.7.1", "google-gax": "^1.14.2", "graphql": "^14.3.1", "handlebars": "^4.7.7", "joi": "^14.3.1", "joi-date-extensions": "^1.2.0", "jsonwebtoken": "^8.3.0", "knex": "^0.21.1", "libphonenumber-js": "^1.7.51", "lodash": "^4.17.21", "merge-graphql-schemas": "1.5.1", "moment": "^2.29.4", "moment-timezone": "^0.5.27", "pg": "^8.7.3", "promise.allsettled": "^1.0.2", "redis": "^3.1.1", "slonik": "^22.4.4", "source-map-support": "^0.5.19", "twilio": "^3.66.1", "ua-parser-js": "^0.7.24", "uuid": "^8.3.0", "winston": "^3.2.1"

DATADOG TRACER CONFIGURATION(2.12.2) { "date": "2022-08-08T::.Z", "os_name": "Linux", "os_version": "5.4.188+", "architecture": "x64", "version": "2.12.2", "lang": "nodejs", "lang_version": "16.15.0", "env": "", "service": "ww", "agent_url": "http://*.*.*.*:8126", "debug": true, "sample_rate": 1, "sampling_rules": [], "tags": { "service": "ww", "env": "", "version": "..*", "runtime-id": "**" }, "dd_version": "..", "log_injection_enabled": true, "runtime_metrics_enabled": false, "profiling_enabled": true, "integrations_loaded": [], "appsec_enabled": false }

DATADOG TRACER CONFIGURATION(0.36.6) { "date": "2022-08-08T::.Z", "os_name": "Linux", "os_version": "5.4.188+", "architecture": "x64", "version": "0.36.6", "lang": "nodejs", "lang_version": "16.15.0", "env": "", "enabled": true, "service": "", "agent_url": "http://*.*.*.*:8126", "debug": true, "analytics_enabled": true, "sample_rate": 1, "sampling_rules": [], "tags": { "service": "", "env": "", "version": ".." }, "dd_version": "..", "log_injection_enabled": true, "runtime_metrics_enabled": false, "integrations_loaded": [ "bluebird@3.7.2", "express@4.17.1", "@google-cloud/pubsub@2.11.0", "graphql@14.7.0", "@grpc/grpc-js@1.3.0", "knex@0.21.5", "redis@3.1.1" ] }

[spaiz]

In our case, we did downgrade to the version 2.1.1, otherwise we have missing tracing breakdown. Not sure if I need open another issue here. As I see, many people have this kind of issues.

We use:

Node v12.16.2

After the downgrade, tracing breakdown back

[rochdev]

Both issues seem like a load order issue. Can you share how the tracer is initialized? It must be initialized before any other imports to work properly. If that's not the case, then there are no guarantees so things might break between versions. More information can be found in the setup docs.

[luyi0521]

@rochdev

FYI.

I think it is right.

./server.ts

// line 1~3 
// other tracers
import other_tracers;
// line 4~16
// check env & initialized dd-tracer
// we only start datadog tracer when code run in cloud platform
if(process.env.NODE_ENV !=='local'){
    import('./tracer').then(tracer => {
        log.log('warn', `Server initiated DD ${tracer}`);
    });
}
// ... other code

./tracer.ts

import tracer from 'dd-trace';
tracer.init();
export default tracer;

[rochdev]

@luyi0521 Dynamic import is not supported, so a static import should be used instead. You could either wrap only the init, or set DD_TRACE_ENABLED=false externally for that environment.

It's also unclear if the imports are transpiled to require calls or if you are using ESM, but for the latter you would also need to use our loader with node --loader=dd-trace/loader-hook.mjs server.

[luyi0521]

Fixed. Thanks a lot.

solution code:

./server.ts

// other tracer
import('./tracer');

// ... other code ...

./tracer.ts

import tracer from 'dd-trace';
if (process.env.NODE_ENV !== 'local') {
    tracer.init(); // initialized in a different file to avoid hoisting
}

export default tracer;

[luyi0521]

@rochdev one more question. dd-trace@2.12.2 can upload tracing & profiling data now. But the Startup log still not contain "integrations_loaded" list as 0.* version like following log. Is it correct?

DATADOG TRACER CONFIGURATION -{
  "date": "2022-08-12T10:30:23.181Z",
  "os_name": "Linux",
  "os_version": "5.4.188+",
  "architecture": "x64",
  "version": "2.12.2",
  "lang": "nodejs",
  "lang_version": "16.15.0",
  "env": "**",
  "service": "**",
  "agent_url": "http://**:8126",
  "debug": false,
  "sample_rate": 1,
  "sampling_rules": [],
  "tags": {
    "service": "w**w",
    "env": "**",
    "version": "*.*.*",
    "runtime-id": "<uuid>"
  },
  "dd_version": "*.*.*",
  "log_injection_enabled": true,
  "runtime_metrics_enabled": false,
  "profiling_enabled": true,
  "integrations_loaded": [],
  "appsec_enabled": false
}

[rochdev]

I think this is a bug with some recent changes to the plugin system which have been fixed but not yet released. Let's keep this issue opened for now but it should work in the next release.

[khanayan123]

Hi @luyi0521 is this still an issue on the latest dd-trace v4 release?

[khanayan123]

Closing due to inactivity, if you are still having this problem please feel free to reopen