Tag found exceptions

[aclowkey]

Can we tag messages which had exceptions detected and not just concatenate ? This would be helpful to change severity to error.

StackDriver error reporting doesn't catch my exception messages, I assmue it's because they are not errors

[igorpeshansky]

Yes, Stackdriver Error Reporting will only look at error logs. Normally, exceptions get printed to stderr, which is ingested as error by default, so it works out. That said, this sounds like a reasonable feature request to me. Once @qingling128 also confirms, we would be happy to review a PR.

[aclowkey]

My application's logs are printed to stderr, but they are run via systemd, which just redirects them to stdout . And, fluentd ingests the logs from systemd.

[igorpeshansky]

While we're reviewing the PR, a workaround would be to add a "severity" entry to your application's log records via some filter plugin (e.g., record_transformer/record_modifier) — see the logging agent documentation.

[aclowkey]

I tested it and its still not being picked up by error repoeting. Perhaps its be aide the stack trace doesn't have new lines?

On Sun, Mar 3, 2019, 18:32 igorpeshansky notifications@github.com wrote:

While we're reviewing the PR, a workaround would be to add a "severity" entry to your application's log records via some filter plugin (e.g., record_transformer/record_modifier) — see the logging agent documentation https://cloud.google.com/logging/docs/agent/configuration#special-fields .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions/issues/54#issuecomment-469039105, or mute the thread https://github.com/notifications/unsubscribe-auth/AB9y2YDa2HITDBS7xAwaz_Rb6z2cZkQ9ks5vS_kUgaJpZM4bW0nx .

[aclowkey]

@igorpeshansky Any progress with the review? This would greatly assist with the error reporting. At this moment, we relay on having loggers which will report the priority properly and we reached already systems where there is no such support

[aclowkey]

This would also allow to take advantage of the type - IDK when this was added, just saw it. https://cloud.google.com/error-reporting/docs/formatting-error-messages#log-entry-examples

{...
  {
    "jsonPayload": {
      **"@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",**
      "message": "An error message without stacktrace or exception"
    },
    "logName": "projects/test-project/logs/reported-error",
    "resource": {
      "labels": {
        "project_id": "test-project"
      },
      "type": "global"
    },
    "severity": "ERROR",
    "timestamp": "2019-06-27T13:43:26.375834551Z"
  }
}

[aclowkey]

The situation is even worse as far as error reporting goes: The stack traces is simply joined in a single line which the error reporting doesn't recognized.

I've set the aforementioned @type parameter, and I got error reported, but "No parsed stack trace available "

Should we joined by new_line? allow configurable delimiter? or flag if to join by new line?

[igorpeshansky]

@aclowkey Does systemd strip newlines? This plugin is designed for processing text logs from files, which do have trailing newlines, so joining them results in a single multiline message. It's also designed for environments where users can't easily access fluentd configuration. Once you can configure fluentd, there are plenty of other ways to transform incoming logs.

I would suggest that you first test this out manually with some specific log entries (see https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions/pull/43#issuecomment-520042624 for instructions on how to do this). Once you've arrived at the format that is handled correctly by the Error Reporting backend, we can see how to achieve that format (which may or may not involve this plugin).

[aclowkey]

I'm not sure if systemd strips them out, or the logger just don't add them.

I read the description which doesn't mention any of these design goals..

Text log messages are assumed to contain single lines and are combined by concatenating them.

and

For JSON records, ..The message field is assumed to contain a single line of text Should you add " that is delimited by \n"?

I could add a new line to all records, but not all of them need it...

[taraspos]

To be able to mark caught exception would be extremely useful. So something like add_tag_prefix, but only to the matched events would be very useful, so later we can route the exception to some other destination (like record_transformer/record_modifier plugins).

I would imagine it to be configured something like:

<match **>
  @type detect_exceptions
  remove_tag_prefix foo
  message log
  languages java, python
  multiline_flush_interval 0.1

  #### start of the new sub-configuration 
  <exception>
       remove_tag_prefix foo
       remove_tag_suffix bar
       add_tag_prefix foo
       add_tag_suffix bar
  </exception>
  #### start of the new sub-configuration 
</match>

where optional subconfiguration exception indicates what to do with the matched exception record