Some of the JS Exceptions and Errors are listed in the "logs" kind instead of "exception"

[mindcurv-jerald]

Description

Some of the JS Exceptions and Errors are listed in the "logs" kind instead of "exception" and the stacktrace is not parsed.

Example logs:

timestamp="2024-02-27 12:54:23.473 +0000 UTC" kind=log message="{\"timestamp\":\"2024-02-27T12:54:23.473Z\",\"app\":\"testing\",\"level\":\"error\",\"message\":\"A valid access token is needed.\",\"stack\":\"Error: A valid access token is needed.\\n at F.prepareRequest (https://testing.com/_app/immutable/chunks/validation.zkRrGPUo.js:1:2526)\\n at F.prepareRequest (https://test-cdn.com/_app/immutable/chunks/cart-store.Uyt4l-39.js:1:1158)\\n at F.request (https://test-cdn.com/_app/immutable/chunks/validation.zkRrGPUo.js:1:2247)\\n at F.getCarts (https://test-cdn.com/_app/immutable/chunks/cart-store.Uyt4l-39.js:1:1525)\\n at Object.w [as loadCarts] (https://test-cdn.com/_app/immutable/chunks/cart-store.Uyt4l-39.js:1:5363)\\n at y (https://test-cdn.com/_app/immutable/nodes/0.kGU01wy8.js:17:13760)\\n at c.invokeTask (https://test-cdn.com/_app/immutable/entry/app.5HXs26dt.js:42:7056)\\n at s.runTask (https://test-cdn.com/_app/immutable/entry/app.5HXs26dt.js:42:2446)\\n at l.invokeTask [as invoke] (https://test-cdn.com/_app/immutable/entry/app.5HXs26dt.js:42:8138)\\n at f (https://test-cdn.com/_app/immutable/entry/app.5HXs26dt.js:67:737)\"} Error: A valid access token is needed." level=error sdk_version=1.3.8 app_name=testing app_version=3b9e0f3 app_environment=stage user_id=37b6c533-20eb-4859-8619-190b6d7e2080 session_id=d2ipdMy0pc session_attr_previousSession=NJhNmgsQvT session_attr_serverSpanId=d45d26bc1c4da253 session_attr_serverTraceId=95ff58bbcd18548901c464b22aab21cc page_url=https://testing.com/de-de/flaechen/flaechenelemente browser_name=Chrome browser_version=122.0.0.0 browser_os="Mac OS 10.15.7" browser_mobile=false view_name=default

Expected behavior

The errors with stacktrace should be listed in the kind exception instead of logs and should have a proper stacktrace

Actual behavior

The errors with stacktrace are listed in the logs and the stacktrace is not parsed.

Environment

• *SDK version:1.3.8*
• *SDK instrumentations:Web SDK, Web Tracing*

[codecapitano]

HI @mindcurv-jerald

By default Faro auto captures unhandled exceptions and rejected promises which do not have rejection handlers attached. These are always captured as an Exception (kind=exception) event.

Faro also instruments the browser console by default and auto captures messages printed to the console, like console.error. These are sent as kind=log.

This causes confusion quite often and happens mostly because of usage of console.error in the code (often in global event handlers).

A simple way to find out would we to temporarily disable the console instrumentation and see if you still see those logs.

Example:

initializeFaro({
  // ...
  instrumentations: [
    ...getWebInstrumentations({
      // Optional, if you want to disable the console instrumentation
      captureConsole: false,

      // Optional, if you want to collect all levels
      captureConsoleDisabledLevels: [],

      // Optional, if you want to disable only some levels
      captureConsoleDisabledLevels: [LogLevel.DEBUG, LogLevel.TRACE],
    }),
  ],
});

Cheers, Marco

[mindcurv-jerald]

Hello @codecapitano

Thanks for your message. I have one more question. I'm not getting the stacktrace properly parsed using the sourcemaps. How do I fix or achieve this ?

I reproduced the following exception which is displayed in the stacktrace field: Error: No CAD format found at O (https://test-cdn.abc.com/test/_app/immutable/chunks/webgl.BcaIGrdQ.js:4:63687)

I get the following error message in Grafana agent when I try to reproduce an exception.

ts=2024-02-29T11:05:09.611898051Z level=error msg="Error resolving stack trace frame source location" component=faro.receiver.default exporter=logs err="unexpected status 404"

Grafana agent flow config have the following configuration :

config.river: |+ // Logging settings logging { level = "info" format = "logfmt" }

// Kubernetes discovery components
discovery.kubernetes "pods" {
  role = "pod"
}

discovery.kubernetes "nodes" {
  role = "node"
}

discovery.kubernetes "services" {
  role = "service"
}

discovery.kubernetes "endpoints" {
  role = "endpoints"
}

discovery.kubernetes "endpointslices" {
  role = "endpointslice"
}

discovery.kubernetes "ingresses" {
  role = "ingress"
}

// Faro receiver input
faro.receiver "default" {
    server {
        api_key = env("GRAFANA_AGENT_FARO_API_KEY")
        listen_address = "0.0.0.0"
        listen_port = 12347
        cors_allowed_origins = ["*"]
    }

    sourcemaps {
        download = true
        location {
            path                 = "/usr/src/app/dist/client/sourcemaps"
            minified_path_prefix = "https://test-cdn.abc.com"
        }
    }

    output {
        logs   = [loki.process.labels.receiver]
        traces = [otelcol.exporter.otlp.traces.input]
    }
}

// Loki output
loki.write "logs" {
    endpoint {
        url = "http://loki-gateway.loki/loki/api/v1/push"
    }
}

// Tempo trace output
otelcol.exporter.otlp "traces" {
  client {
    endpoint = "tempo-distributor.tempo:4317"
    tls {
      insecure =true
    }
  }
}

// Tempo trace input
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "127.0.0.1:4317"
  }
  http {
    endpoint = "127.0.0.1:4318"
  }
  output {
    metrics = [otelcol.exporter.otlp.traces.input]
    logs    = [otelcol.exporter.otlp.traces.input]
    traces  = [otelcol.exporter.otlp.traces.input]
  }
}

`

The path /usr/src/app/dist/client/sourcemaps is defined as the sourcemap path in the grafana agent configuration. This is the correct path of sourcemaps in the frontend application filesystem and not grafana agent. Can you help here on how to define the path properly ?

[kminehart]

At the moment, source maps are only used if a comment with the format of //# sourceMappingURL=<url> is set in the source file that is throwing the exception. The source map must also be publicly downloadable.

We are currently working on making it possible to upload your own source maps.

[mindcurv-jerald]

Hi @kminehart In our case mentioned commit is already set in the source file. but the sourcemap location is different and not stored in the same path as the source file, its stored in another path and exposed to public with /sourcemaps path with an authentication.

Since I did not find an option to provide authentication option in the sourcemap path I was exploring the location block mentioned in the below documentation.

https://grafana.com/docs/agent/latest/flow/reference/components/faro.receiver/#location-block.

    sourcemaps {
        download = true
        location {
            path                 = "/usr/src/app/dist/client/sourcemaps"
            minified_path_prefix = "https://test-cdn.abc.com"
        }
    }

I thought the path in the location block here is the path in the filesystem of the frontend app. and minified_path_prefix is the minified URL from the stack trace. Is this not the case ?

If not is there a way to use sourcemap publicly with authentication ?

[kminehart]

Ah OK the agent flow component is a bit new to me. I was thinking that you were sending data straight to our API endpoint.

It looks to me like the Grafana Agent will attempt to translate the exception before it gets sent to our remote receiver using the settings in that block. You can see the logic here: https://github.com/grafana/agent/blame/0aeca5963d82d63be1c596d210bdc53a44959f27/component/faro/receiver/sourcemaps.go

It follows a similar logic to what I said earlier; it downloads and prases the source file looking for the sourceMappingURL. What is that sourceMappingURL value in your source file?

The path /usr/src/app/dist/client/sourcemaps is defined as the sourcemap path in the grafana agent configuration. > This is the correct path of sourcemaps in the frontend application filesystem and not grafana agent.

If you want to use a source map on the filesystem, then make sure the URL in that comment is formatted with data:///path/to/sourcemap

[mindcurv-jerald]

@kminehart Thank you for your response.

The current path at the end of the sourcefile URL is //# sourceMappingURL=webgl.BcaIGrdQ.js.map

Path of the sourcemap in the app filesystem is /usr/src/app/dist/client/sourcemaps and path of the source JS files are /usr/src/app/dist/client/_app/

Should the sourcemap path in the source JS file be changed to data:///usr/src/app/dist/client/sourcemaps/webgl.BcaIGrdQ.js.map ?

[kminehart]

It's worth a shot. I'll take a closer look at that go file and see if I have any other suggestions.

[kminehart]

hmm actually it seems like what you have should work. Is webgl.BcaIGrdQ.js.map in /usr/src/app/dist/client/sourcemaps?

I thought I read in one of your comments that the source map was not on the filesystem where the agent is running. That'd be a problem in this set up. I think we may need a feature request to add basic authentication / http header options to the sourcemaps block.

[mindcurv-jerald]

Hi @kminehart

The path was initially wrong, I somehow got the sourcemap path working by specifying the location block as described in page: https://grafana.com/docs/agent/latest/flow/reference/components/faro.receiver/#location-block

Additionally I pushed the sourcemaps to a storage and mounted the storage in grafana agent to get the sourcemaps to work.

    sourcemaps {
        download = true
        location {
            path                 = "/mnt/testing/dev/sourcemaps/dist/client/_app/"
            minified_path_prefix = "https://dev-cdn.abc24.com/testing/_app/"
        }
    }

Stacktrace Error without sourcemaps:

Error: just a test
  at HTMLButtonElement.T (https://dev-cdn.abc24.com/testing/_app/immutable/nodes/15.09bflFDK.js:5:281)
  at ? (https://dev-cdn.abc24.com/testing/_app/immutable/chunks/scheduler.1hU0N1-4.js:1:9659)
  at Array.forEach (<anonymous>:0:0)
  at HTMLButtonElement._e (https://dev-cdn.abc24.com/testing/_app/immutable/chunks/scheduler.1hU0N1-4.js:1:9646)
  at HTMLButtonElement.me (https://dev-cdn.abc24.com/testing/_app/immutable/chunks/button.up6xjsjg.js:1:9726)
  at ? (https://dev-cdn.abc24.com/testing/_app/immutable/chunks/scheduler.1hU0N1-4.js:1:9659)
  at Array.forEach (<anonymous>:0:0)
  at HTMLButtonElement._e (https://dev-cdn.abc24.com/testing/_app/immutable/chunks/scheduler.1hU0N1-4.js:1:9646)
  at HTMLButtonElement.b (https://dev-cdn.abc24.com/testing/_app/immutable/chunks/button.up6xjsjg.js:1:1736)
  at c.invokeTask (https://dev-cdn.abc24.com/testing/_app/immutable/entry/app.mUUx7avW.js:42:7056)

Stacktrace Error with sourcemaps:

Error: just a test
  at ? (https://dev-cdn.abc24.com/src/routes/%5Blocale%5D/test/+page.svelte:5:14)
  at ? (https://dev-cdn.abc24.com/node_modules/svelte/src/runtime/internal/lifecycle.js:181:39)
  at Array.forEach (<anonymous>:0:0)
  at ? (https://dev-cdn.abc24.com/node_modules/svelte/src/runtime/internal/lifecycle.js:181:20)
  at $$props (https://dev-cdn.abc24.com/src/lib/common/components/button/button.svelte:23:48)
  at ? (https://dev-cdn.abc24.com/node_modules/svelte/src/runtime/internal/lifecycle.js:181:39)
  at Array.forEach (<anonymous>:0:0)
  at ? (https://dev-cdn.abc24.com/node_modules/svelte/src/runtime/internal/lifecycle.js:181:20)
  at $$props (https://dev-cdn.abc24.com/src/lib/common/components/button/button-content.svelte:4:53)
  at ? (https://dev-cdn.abc24.com/node_modules/zone.js/fesm2015/zone.js:406:30)

As you can see that there is question mark sign "?" in the parsed stacktrace. Can you help me here to know what does the question mark indicate and how to fix it ?

[mindcurv-jerald]

HI @mindcurv-jerald

By default Faro auto captures unhandled exceptions and rejected promises which do not have rejection handlers attached. These are always captured as an Exception (kind=exception) event.

Faro also instruments the browser console by default and auto captures messages printed to the console, like console.error. These are sent as kind=log.

This causes confusion quite often and happens mostly because of usage of console.error in the code (often in global event handlers).

A simple way to find out would we to temporarily disable the console instrumentation and see if you still see those logs.

Example:

initializeFaro({
  // ...
  instrumentations: [
    ...getWebInstrumentations({
      // Optional, if you want to disable the console instrumentation
      captureConsole: false,

      // Optional, if you want to collect all levels
      captureConsoleDisabledLevels: [],

      // Optional, if you want to disable only some levels
      captureConsoleDisabledLevels: [LogLevel.DEBUG, LogLevel.TRACE],
    }),
  ],
});

Cheers, Marco

@codecapitano

is it possible to analyze log messages for stack traces as well, and not just uncatched exceptions? Or does this only work if the message itself also uses the logfmt format instead of json?

[codecapitano]

Hi @mindcurv-jerald

is it possible to analyze log messages for stack traces as well, and not just uncatched exceptions? Or does this only work if the message itself also uses the logfmt format instead of json?

For checked exceptions you can use the Faro pushError api which parses the stack-trace. Faro can not derive stacktraces from log messages, it needs proper js error objects.

See: When to report errors manually

// Supposed this error is generated somewhere in your app
const error = new Error('I am supposed to fail');
// ...
// And at some point you want to pipe it to Faro
faro.api.pushError(error);

Cheers, Marco

[mindcurv-jerald]

Thank you @codecapitano

I have created a feature request https://github.com/grafana/faro-web-sdk/issues/509 to enable authentication option within the sourcemap block.