[Bug] <unlabeled event>

[marvinosswald]

i've seen a few discussions regarding <unlabeled event> but the following body seems fine by the looks of it, doesn't it ?

We are seeing this mostly on the following Client Setup: IE 11.0 Windows (mostly 7)

{
  "name": "exception",
  "value": {
    "values": [
      {
        "stacktrace": {
          "frames": [
            {
              "function": "?",
              "filename": "https://mysvg.de/",
              "in_app": true
            }
          ]
        },
        "value": {}
      }
    ]
  }
}

Our Angular 4 ErrorHandler Class:

export class GlobalErrorHandler implements ErrorHandler {
    static isErrorOrErrorEvent (wat) {
      return Object.prototype.toString.call(wat) === '[object Error]' || Object.prototype.toString.call(wat) === '[object ErrorEvent]';
    }
    constructor(private injector: Injector) {}

    handleError(err) {
      const error = err.originalError || err;

      if (GlobalErrorHandler.isErrorOrErrorEvent(error)) {
        Raven.captureException(error);
      } else {
        Raven.captureMessage(error.error || error);
      }
    }
}

[sigurbjorngudjonsson]

I'm experiencing the same on an alternative javascript stack, using React with Redux and Redux Sagas, seeing the same error being reported both labeled and unlabeled.

Some noticeable differences are between the JSON structures of the issues:

Labeled:

{
  "errors": [],
  "sentry.interfaces.Exception": { ... }
}

Unlabeled:

• Missing the "sentry.interfaces.Exception": { ... } part

  {
  "errors": [{
    "type": "invalid_data",
    "name": "exception",
    "value": {
      "values": [
        {
          "stacktrace": {
            "frames": [...]
          },
          "type": "TypeError",
          "value": "Some error message"
        }
      ]
    }
  }],
  }

Wandering what could yield such different results? 🤔

[shcallaway]

My team is experiencing this with Angular 5 on IE 11. Seems related to the following:

• Why all my events are ‘unlabeled event’ and have Discarded invalid value for exception and breadcrumb?
• Lots of \<unlabeled event> events

It sounds like it may be related to the format and/or type of the values field.

@marvinosswald By the way, I think the ErrorEvent issue you are solving in your error handler was resolved as part of this release.

[kamilogorek]

As Ben mentioned in one of the linked issues above, I'm not able to fix this issue unless I can reproduce it. I tried to use quick start example from Angular 4, Angular 5 and Ember and I wasn't able to trigger this issue.

Unless someone is able to provide a reproducible code configuration, it's just a guesswork and probably won't be effective at all.

And yes, ErrorEvent is fixed now. You can pass it directly to captureException and we'll handle it ourselves.

[shcallaway]

For the record, with all my unlabeled events, an error occurs during processing:

[kamilogorek]

@shcallaway can you post a link to this event? I'll try to track it this way

[shcallaway]

I got around to digging deeper into this, and I think I found the source of the problem.

Where does “unlabeled event” come from?

<unlabeled event> is not something that users are passing, nor is it something that Raven applies. Where does it come from?

The answer is the Sentry server:

    def get_metadata(self):
        # See GH-3248
        message_interface = self.data.get(
            'sentry.interfaces.Message', {
                'message': self.data.get('message', ''),
            }
        )
        message = strip(message_interface.get('formatted', message_interface['message']))
        if not message:
            title = '<unlabeled event>'
        else:
            title = truncatechars(message.splitlines()[0], 100)
        return {
            'title': title,
        }

https://github.com/getsentry/sentry/blob/96bc1c63df5ec73fe12c136ada11561bf52f1ec9/src/sentry/eventtypes/base.py#L38

When handling a DefaultEvent object, Sentry attempts to get the “sentry.interfaces.Message” property (unsuccessfully) and falls back to the default value: { 'message': self.data.get('message', '') }

Which in turn evaluates to: { ‘message': ‘’ }

So message_interface is assigned to this object, and on line 36, message is assigned to message_interface[‘message’] — an empty string!

In the conditional on line 37, the empty string evaluates to false and title is assigned to our old friend, <unlabeled event>.

TL;DR — The DefaultEvent is lacking both of the following properties: sentry.interfaces.Message and message. So Sentry uses a stand-in value as the title: <unlabeled event>

There's an error while processing this event. Why?

All unlabeled event errors appear alongside the same message: “There was 1 error encountered while processing this event”.

I believe this is the line responsible for adding the processing error.

The code appears to be casting each property (environment, breadcrumbs, exception, etc.) to a specific format. When the cast fails, it adds a processing error.

What happens next is highly interesting: The raw ‘message’ is coerced to the Sentry message interface:

        # raw 'message' is coerced to the Message interface, as its used for pure index of
        # searchable strings. If both a raw 'message' and a Message interface exist, try and
        # add the former as the 'formatted' attribute of the latter.
        # See GH-3248
        msg_str = data.pop('message', None)
        if msg_str:
            msg_if = data.setdefault('sentry.interfaces.Message', {'message': msg_str})
            if msg_if.get('message') != msg_str:
                msg_if.setdefault('formatted', msg_str)

If the message property is missing, it defaults to None, and msg_str becomes falsy. If msg_str is falsy, the sentry.interfaces.Message and formatted properties are never assigned to msg_if. As we saw at the beginning of this post, these properties are necessary to prevent title from being assigned to <unlabeled event>.

Hypothesis

Raven is sending a bad message object to the Sentry server. Specifically, it is missing the message property — or the message property is a falsy value, such as an empty string.

My guess is that somewhere in the captureException function (here), captureMessage is being passed an empty string for the msg param. Unfortunately, I can't figure out which if statement the error falls into because don't have access to the original error object —and I don't know how to reproduce it.

Solution

We need to make sure to call captureMessage with a good msg param. In other words, we should not be passing it an empty string.

captureMessage is being called within captureException, so it looks like there's an error type (or object shape) that we are not accounting for there.

[shcallaway]

@kamilogorek Can I get your thoughts on this?

[kamilogorek]

@shcallaway I'm working on something else right now, but I'll get back to you early next week. Thanks for very detailed investigation! :)

[justinappler]

Still seeing this hundreds, sometimes thousands of times per hour but we can't hide it because don't know if its a real issue or noise.

[justinappler]

@kamilogorek Any update? This represents a huge number of our inbound exceptions and it's hard for us to ignore it given we don't know if it masks real issues or not.

[kamilogorek]

After some research, I found that ways to trigger this issue are:

captureException('');
captureMessage('');
Raven.wrap(function () {
  throw '';
});

to trigger the exact issue, including mentioned server validation notification, throw '', but it has to bubble up to the global on-error handler.

I'm working on the solution.

[kamilogorek]

@justinappler @shcallaway do you have some example events that I could take a look at by any chance? Any link to an event on your's organization sentry.io account will work.

[justinappler]

@kamilogorek Here's the one we're seeing: https://sentry.io/crunchbase/client-app-v3-production/issues/364876156/?query=is:unresolved%20timesSeen:%3E5

[glebmachine]

Got about 39k events from 1.2k users in 2 weeks. If it helps - here is a link: https://sentry.io/beta-digital-production/svo-frontend/issues/561763091/activity/

I have investigated that metrika.yandex.ru (Russian Google Analytics analogue) sometimes raising error which not logged to Sentry (only IE11 on windows7)

Here is a log:

stack   "Error: Invalid argument.
at z (https://mc.yandex.ru/metrika/tag.js:65:441)
at Anonymous function (https://mc.yandex.ru/metrika/tag.js:284:289)
at b.default.reduce (https://mc.yandex.ru/metrika/tag.js:187:71)
at value (https://mc.yandex.ru/metrika/tag.js:284:215)
at Anonymous function (https://mc.yandex.ru/metrika/tag.js:283:301)
at b.default.reduceRight (https://mc.yandex.ru/metrika/tag.js:187:220)
at value (https://mc.yandex.ru/metrika/tag.js:283:245)
at t.prototype.invoke (http://new.svo.aero/polyfills.ae8cfcafa087f578b772.js:1:22818)
at n.prototype.runGuarded (http://new.svo.aero/polyfills.ae8cfcafa087f578b772.js:1:18263)
at Anonymous function (http://new.svo.aero/polyfills.ae8cfcafa087f578b772.js:1:17892)"

[kamilogorek]

Released as 3.26.3. Would appreciate a feedback with everyone's findings.

[shcallaway]

unlabeled event has totally dropped off for us. Thanks!

[kamilogorek]

Glad to hear that, thanks @shcallaway!

[glebmachine]

The same, we've got about 5 event per week (instead of thousands)

[justincy]

We just turned on Sentry and are seeing lots of <unlabeled event> errors https://sentry.io/familysearchorg/fs-home-prod/issues/735066771/events/33764492600/

Looks like we're using v4.0.4

[kamilogorek]

@justincy can you bump to 4.1.1 and see if that helps? We patched a lot of things since initial release.

One of this things could resolve your issue:

[browser] fix: Check for anonymous function before getting its name for mechanism [browser] fix: Fallback to Error object when rejection reason is not available [browser] feat: Better mechanism detection in TraceKit

[justincy]

We're using the new Loader which is supposed to automatically give us the latest. I guess that's not happening?

[kamilogorek]

@justinappler yes, this is a huge oversight on our side. Fixing it as we speak. Will get back to you once it's live.

[justincy]

@kamilogorek Do you have an ETA for the fix?

[HazAT]

@justinappler It's live right now

[justincy]

Thanks. I see now that we're getting reports from v4.2.3 of the SDK but unfortunately we're still seeing <unlabeled event>

[HazAT]

@justincy can you make sure at the bottom of the event (in the SDK section) that it says 4.2.3 or even better you send me a link to one of those events to daniel@sentry.io

[kamilogorek]

@justincy this one will be very hard to track from what I can see. You are using dynamically loaded web-components, which are built on top of Polymer.

You are loading html files, which are on the other hand loading a) external JS files and b) calling Githubissues.

Githubissues is a development platform for aggregating issues.