search

microsoft / ApplicationInsights-node.js

Microsoft Application Insights SDK for Node.js
MIT License
325 stars 140 forks source link

Correlation between Frontend and Backend. #1333

Open obiwanjacobi opened 6 months ago

obiwanjacobi commented 6 months ago

I have a React app running with @microsoft/applicationinsights-web that makes some HTTP calls to my NodeJS backend (B4F). The BE uses this package. Both are configured pointing to the same Application Insights instance on Azure.

I would like to correlate the FE calling the backend. I have added some headers to the request containing the traceId, sessionId and traceParent (from the FE). If this can be picked up automatically on the BE what do I call these headers in that case. Alternatively what code do I need to write to pass the data onto the BE AppInsights?

The BE also makes other HTTP calls to other systems I would like to correlate as well. I am using trackDependency for that but I am not seeing the 'call sequence' in the AppInsights detail view on the Azure Portal...? I suspect it is missing a session or parent id in order to tie them together. Any hint for that would be great.

obiwanjacobi commented 6 months ago

Hey @JacksonWeber, how is the investigation going?

JacksonWeber commented 5 months ago

Hi @obiwanjacobi, apologies for the delay, finally got some time to break this down.

Looks like there's two questions here:

  1. There shouldn't be any additional code change necessary (no need to manually handle context propagation!) assuming you've instrumented both the frontend and backend applications with application insights SDKs and are trying to track an HTTP/HTTPS call between the two. If you're having trouble setting this up I can provide a sample app that demonstrates the setup.
  2. If you're looking to track HTTP/HTTPS calls to other services, you shouldn't need to utilize the trackDependency method as these calls should be auto-collected and auto-correlated. Since this isn't working for you, could you provide an example of a call to one of these calls to other systems and a copy of your package.json would be helpful. Thanks!
obiwanjacobi commented 5 months ago

1) We use Koa on the BE perhaps that is the issue? An example is very much appreciated. 2) We use fetch. This is an example of our post function. Similar for get, put etc.

private async post(resource: string, requestBody: any, headers: Record<string, string>): Promise<Response> {
        const allHeaders = headers
            ? { ...headers, Authorization: this.authorization }
            : { "Content-Type": "application/json", Authorization: this.authorization };

        const url = `${this.baseUrl}/rest/api/2/${resource}`;
        const start = Date.now();
        const response = await fetch(url, {
            method: "POST",
            headers: allHeaders,
            body: requestBody,
        });

        logRequest("POST", url, response.status, start);    // <= trackDependency

        return response;
    }

We use a mono-repo so the FE and BE is mixed.

package.json:

{
  "name": "<project name>",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "== our scripts ==" : "<removed>"
  },
  "dependencies": {
    "@azure/identity": "^3.4.1",
    "@azure/keyvault-secrets": "^4.7.0",
    "@fluentui/react": "^8.112.0",
    "@fluentui/react-calendar-compat": "^0.1.5",
    "@fluentui/react-components": "^9.46.4",
    "@fluentui/react-datepicker-compat": "^0.4.35",
    "@fluentui/react-icons": "^2.0.218",
    "@koa/cors": "^4.0.0",
    "@koa/router": "^12.0.1",
    "@lexical/react": "0.12.4",
    "@microsoft/applicationinsights-web": "^3.2.1",
    "@nx/webpack": "^16.10.0",
    "@rjsf/core": "^5.14.3",
    "@rjsf/fluentui-rc": "^5.14.3",
    "@rjsf/utils": "^5.14.3",
    "@rjsf/validator-ajv8": "^5.14.3",
    "@swc/helpers": "~0.5.2",
    "@tanstack/react-query": "^5.0.5",
    "@tanstack/react-query-devtools": "^5.4.3",
    "@tanstack/react-table": "^8.11.7",
    "@types/he": "^1.2.3",
    "ajv-errors": "^3.0.0",
    "applicationinsights": "^3.1.0",
    "async-wait-until": "^2.0.12",
    "axios": "^1.0.0",
    "azure-devops-extension-api": "4.229.0",
    "azure-devops-extension-sdk": "4.0.2",
    "azure-devops-ui": "^2.236.0",
    "chalk": "^4.1.2",
    "commander": "^10.0.1",
    "console-table-printer": "^2.11.1",
    "cors": "^2.8.5",
    "export-from-json": "^1.7.4",
    "express": "~4.18.1",
    "figlet": "^1.6.0",
    "he": "^1.2.0",
    "immer": "^10.0.3",
    "koa": "~2.14.1",
    "koa-404-handler": "^0.1.0",
    "koa-better-error-handler": "^11.0.4",
    "koa-body": "^6.0.1",
    "koa-morgan": "^1.0.1",
    "koa-static": "^5.0.0",
    "lexical": "0.12.4",
    "lodash.startcase": "^4.4.0",
    "loglevel": "^1.9.1",
    "markdown-to-jsx": "^7.3.2",
    "morgan": "^1.10.0",
    "node-cache": "^5.1.2",
    "node-fetch": "^2.7.0",
    "parse-json": "^7.0.0",
    "patch-package": "^8.0.0",
    "re-resizable": "^6.9.11",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "react-drawio": "^0.1.2",
    "react-grid-layout": "^1.4.4",
    "react-markdown": "^8.0.7",
    "react-modal": "^3.16.1",
    "react-router-dom": "^6.18.0",
    "shelljs": "^0.8.5",
    "source-map-support": "^0.5.13",
    "tslib": "^2.3.0",
    "typescript-logging": "^2.1.0",
    "typescript-logging-category-style": "^2.1.0",
    "undici": "^5.27.0",
    "uuid": "^9.0.1",
    "vss-web-extension-sdk": "^5.141.0",
    "yaml": "^2.2.2"
  },
  "devDependencies": {
    "@babel/core": "^7.14.5",
    "@babel/preset-react": "^7.14.5",
    "@nrwl/nx": "^7.8.7",
    "@nx/esbuild": "16.10.0",
    "@nx/eslint-plugin": "16.10.0",
    "@nx/jest": "16.10.0",
    "@nx/js": "16.10.0",
    "@nx/linter": "16.10.0",
    "@nx/node": "16.10.0",
    "@nx/playwright": "^16.10.0",
    "@nx/react": "16.10.0",
    "@nx/storybook": "^16.10.0",
    "@nx/vite": "16.10.0",
    "@nx/web": "16.10.0",
    "@nx/workspace": "16.10.0",
    "@playwright/test": "^1.36.0",
    "@storybook/addon-essentials": "7.4.6",
    "@storybook/addon-interactions": "^7.2.1",
    "@storybook/core-server": "7.4.6",
    "@storybook/jest": "~0.1.0",
    "@storybook/manager-api": "^7.4.6",
    "@storybook/react-vite": "7.4.6",
    "@storybook/test-runner": "^0.13.0",
    "@storybook/testing-library": "~0.2.0",
    "@storybook/theming": "^7.4.6",
    "@swc/cli": "^0.1.63",
    "@swc/core": "~1.3.85",
    "@swc/jest": "0.2.20",
    "@testing-library/jest-dom": "^6.1.4",
    "@testing-library/react": "14.0.0",
    "@types/cors": "^2.8.15",
    "@types/decompress": "^4.2.5",
    "@types/express": "~4.17.13",
    "@types/jest": "^29.4.0",
    "@types/koa": "~2.13.5",
    "@types/koa__router": "^12.0.3",
    "@types/lodash.startcase": "^4.4.9",
    "@types/morgan": "^1.9.7",
    "@types/node": "^18.14.2",
    "@types/node-fetch": "^2.6.6",
    "@types/react": "18.2.24",
    "@types/react-dom": "18.2.9",
    "@types/react-grid-layout": "^1.3.5",
    "@types/react-modal": "^3.16.2",
    "@types/shelljs": "^0.8.11",
    "@types/unzip-stream": "^0.3.2",
    "@types/uuid": "^9.0.7",
    "@typescript-eslint/eslint-plugin": "^5.60.1",
    "@typescript-eslint/parser": "^5.60.1",
    "@vitejs/plugin-basic-ssl": "^1.0.1",
    "@vitejs/plugin-react": "~4.0.0",
    "@vitejs/plugin-react-swc": "~3.3.2",
    "@vitest/coverage-c8": "~0.32.0",
    "@vitest/ui": "~0.32.0",
    "ajv": "^8.12.0",
    "babel-jest": "^29.4.1",
    "decompress": "^4.2.1",
    "esbuild": "~0.19.2",
    "eslint": "~8.46.0",
    "eslint-config-prettier": "8.1.0",
    "eslint-plugin-import": "2.27.5",
    "eslint-plugin-jsx-a11y": "6.7.1",
    "eslint-plugin-playwright": "^0.15.3",
    "eslint-plugin-react": "7.32.2",
    "eslint-plugin-react-hooks": "4.6.0",
    "jest": "^29.4.1",
    "jest-environment-jsdom": "^29.4.1",
    "jest-environment-node": "^29.4.1",
    "jsdom": "~22.1.0",
    "kill-port": "^2.0.1",
    "nodemon": "^3.1.0",
    "nx": "16.10.0",
    "otpauth": "^9.1.5",
    "postcss-url": "^10.1.3",
    "prettier": "^2.6.2",
    "react-docgen": "^7.0.3",
    "rollup-plugin-copy": "^3.5.0",
    "sass": "^1.69.0",
    "scheduler": "^0.23.0",
    "tfx-cli": "^0.14.0",
    "tfx-umbrella": "file:./packages/tfx-umbrella",
    "the-new-css-reset": "^1.11.0",
    "ts-jest": "^29.1.0",
    "ts-json-schema-generator": "^1.5.0",
    "ts-node": "10.9.1",
    "typedoc": "^0.25.10",
    "typedoc-plugin-missing-exports": "^2.2.0",
    "typescript": "~5.1.3",
    "unzip-stream": "^0.3.1",
    "verdaccio": "^5.0.4",
    "vite": "~4.5.3",
    "vite-plugin-dts": "~2.3.0",
    "vitest": "~0.32.0"
  },
  "optionalDependencies": {
    "@nx/nx-darwin-arm64": "16.10.0",
    "@nx/nx-darwin-x64": "16.10.0",
    "@nx/nx-linux-x64-gnu": "16.10.0",
    "@nx/nx-linux-x64-musl": "16.10.0",
    "@nx/nx-win32-x64-msvc": "16.10.0"
  },
  "engines": {
    "npm": ">=9.6.7",
    "node": ">=18.17.0"
  }
}
JacksonWeber commented 5 months ago

As for the simple example app, I've attached a zipped project. web-node-correlation-test.zip

Your issue appears to be that you're using the fetch, which we don't support auto-instrumentation for yet. Support for fetch will be coming upon adoption of the fetch instrumentation from upstream OpenTelemetry in the Azure Monitor OpenTelemetry project (which 3.X of this project is a wrapper around).

If you need support for this instrumentation today to allow the use of fetch, you can pair Azure Monitor OpenTelemetry with the community instrumentation for the fetch API here.

JacksonWeber commented 5 months ago

Closing with this comment. Please reopen if you have further questions.

obiwanjacobi commented 5 months ago

I have tried your example but I don't see the frontend and backend calls correlated in AppInsights..? All I see is the backend with it's 'dependency' to bing that is being called. Am I doing something wrong or are my expectations unrealistic?

JacksonWeber commented 5 months ago

@obiwanjacobi Updated the example above to be simpler and more straightforward. Let me take a look regarding the correlation.

JacksonWeber commented 5 months ago

@obiwanjacobi Fixed the issue with the original sample app. You should now see correlated telemetry like this example in the Azure Portal: image

obiwanjacobi commented 5 months ago

I'm sorry but it does not work at my end. I am not sure if I am doing something wrong...

I only see the bing dependency being logged...

JacksonWeber commented 5 months ago

@obiwanjacobi Apologies, looks like there were issues with CORS in the example. I've updated the example to set the correct CORS policy for you in the node-server and updated the frontend HTML to enableCorsCorrelation: true.

Steps are:

  1. run npm install in both the frontend and backend packages
  2. Update both the index.html in frontend and index.js in the node-server folder with your connection string
  3. Navigate into the node-server folder and run node index.js
  4. Navigate into the frontend folder and run node index.js
  5. Navigate to http://localhost:8080 (frontend server address) once both are running
  6. Hit the "Start" button element to send an XMLHTTP request to the node server (which makes the call to the bing.com dependency)

Final result should look like: image

JacksonWeber commented 4 months ago

@obiwanjacobi Let me know if this resolved your issue, thanks!

obiwanjacobi commented 3 months ago

No it did not - but I gave up.

docouto commented 2 months ago

hey @JacksonWeber @obiwanjacobi I want to thank you for the chat on this thread. It helped on my quest!

I found this thread trying to understand how to get the stamp ('Operational ID') that AppInsights put on the requests, the example code you shared helped me to understand the flow between front and back. But I still could not get the information I need.

If I may, I want to share my use case.

I have a front end in React and back end APIs in Node. Those APIs are exposed through Azure APIM. So, something like this: Front (React) -> APIM -> Back (Node).

I would like to get a response header with the operational ID o the request I made on the front end. Using this APIM Policy I could come up with a custom response header with a GUID searchable on logs.

The question is: is it possible to pass to the front end a response header with the operation ID, that AppInsights stamped a request?

Let me know if that is out of scope and I can raise another issue.

Cheers. Denis

obiwanjacobi commented 1 month ago

@obiwanjacobi Let me know if this resolved your issue, thanks!

What call should I use to log into app insight (related to the current call/span) on the frontend and the server? It seems console.log() doesn't cut it.

JacksonWeber commented 1 month ago

@docouto I'm a little confused. Are you trying to propagate context (in this case just via a header) from the Node.js backend -> your React frontend?

JacksonWeber commented 1 month ago

@obiwanjacobi I'm not sure I understand your question. What kind of information are you trying to log, and where are you trying to view those logs?

obiwanjacobi commented 1 month ago

@obiwanjacobi I'm not sure I understand your question. What kind of information are you trying to log, and where are you trying to view those logs?

I want to log custom entries into the call hierarchy I see in App-Insights (where I view these logs).

For instance on the frontend (from the sample) - using console as a log api:

function submit() {

    console.log('this is submit'); // <= custom log entry

    var xhr = new XMLHttpRequest();
    xhr.open('get', 'http://localhost:3000');
    xhr.send();
}

I want to do similar things on the backend too.

Because calls to console.log() do not seem to be intercepted (does not appear in app-insights), what function should I use that my log entry will appear (in this case) just before the call to the backend in app-insights?

The @microsoft/applicationinsights-web we were using had the trackTrace() and trackEvent() functions.

obiwanjacobi commented 1 month ago

Hey @JacksonWeber,

Can you explain how I can obtain the sdk instance? I have tried using an onInit function to grab the sdk instance, but even returning false (or void) screws up the default configuration and I do not get the app insights output I am looking for. Not using an onInit callback does give me the output I need.

I have also tried to return the sdk-instance from the - minified - head-script function (variable i seems to be what is passed to the onInit callback) but that also didn't work. Good chance I screwed that up because that code is seriously hard to grasp.

I would prefer to have an onInit callback and still use the default initialization...

JacksonWeber commented 1 month ago

@obiwanjacobi I'm not sure I understand your question. What kind of information are you trying to log, and where are you trying to view those logs?

I want to log custom entries into the call hierarchy I see in App-Insights (where I view these logs).

For instance on the frontend (from the sample) - using console as a log api:

function submit() {

console.log('this is submit'); // <= custom log entry

var xhr = new XMLHttpRequest();
xhr.open('get', 'http://localhost:3000');
xhr.send();

} I want to do similar things on the backend too.

Because calls to console.log() do not seem to be intercepted (does not appear in app-insights), what function should I use that my log entry will appear (in this case) just before the call to the backend in app-insights?

The @microsoft/applicationinsights-web we were using had the trackTrace() and trackEvent() functions.

Please read https://github.com/microsoft/ApplicationInsights-node.js?tab=readme-ov-file#configuration - as it mentions you need to explicitly enable collection of console.log calls.

JacksonWeber commented 1 month ago

Hey @JacksonWeber,

Can you explain how I can obtain the sdk instance? I have tried using an onInit function to grab the sdk instance, but even returning false (or void) screws up the default configuration and I do not get the app insights output I am looking for. Not using an onInit callback does give me the output I need.

I have also tried to return the sdk-instance from the - minified - head-script function (variable i seems to be what is passed to the onInit callback) but that also didn't work. Good chance I screwed that up because that code is seriously hard to grasp.

I would prefer to have an onInit callback and still use the default initialization...

I'm assuming this question is about the web SDK, in which case it'd be better to raise it on that repository. Happy to discuss further if I'm mistaken.

obiwanjacobi commented 1 month ago

Please read https://github.com/microsoft/ApplicationInsights-node.js?tab=readme-ov-file#configuration - as it mentions you need to explicitly enable collection of console.log calls.

I am confused @JacksonWeber

I thought the ApplicationInsights-node package is for the (node) backend/server. I am talking about the frontend - where we used to use the ApplicationInsights-web package.

Another question: that minified head-script, is that compatible with the ApplicationInsights-web package, or is that specific? (I would rather use an official npm package than some minified script magic)

I'm assuming this question is about the web SDK, in which case it'd be better to raise it on that repository.

What repo is that? (like I said, I am confused about all these packages and when/where to use them)

JacksonWeber commented 1 month ago

@obiwanjacobi You're correct that this package (ApplicationInsights-node) is not compatible with web JS, which is why I mentioned it would make more sense to raise frontend JS related issues against the web JS Application Insights SDK.

The basic rule for usage between the usage of these two SDKs is just that if you're writing Node.js backend/server code, expect to be able to instrument using this SDK, whereas if you're running JS in the browser, use the JS SDK linked above.