search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.9k stars 2.27k forks source link

[pkg/stanza] Operators configured with Silent Errors would still result in error logs #35008

Closed SamerJ closed 1 week ago

SamerJ commented 1 week ago

Component(s)

pkg/stanza, pkg/stanza/fileconsumer, receiver/filelog

What happened?

Description

Operators have a config that allow silencing errors logs (log level is changed from error to debug). With this configuration, we expect no error logs when the operator fails to process a log entry. However, even when we use drop_quiet, we still see error logs when the Operator fails. The error is not emitted by the Operator but by another component. Nevertheless, the error is because of the operator's log entry failure.

Steps to Reproduce

Use any operator, that might sometimes fails like regex with a pattern that might not always match. Set the property on_error: drop_quiet. Feed the operator log entries were some are expected to fail the operator. Check the emitted logs.

Expected Result

No errors logs when it comes to failures by the Operator

Comparison

You can see that with quiet we hide some of the error logs but not all.

Default

2024-09-04T16:41:58.270+0200 ERROR helper/transformer.go:100 Failed to process entry {"operator_id": "regex_parser", "operator_type": "regex_parser", "error": "regex pattern does not match", "action": "send"} github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(TransformerOperator).HandleEntryError C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/operator/helper/transformer.go:100 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(ParserOperator).ParseWith C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/operator/helper/parser.go:138 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(ParserOperator).ProcessWithCallback C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/operator/helper/parser.go:111 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(ParserOperator).ProcessWith C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/operator/helper/parser.go:98 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(Parser).Process C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/operator/parser/regex/parser.go:31 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(WriterOperator).Write C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/operator/helper/writer.go:54 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/filter.(Transformer).Process C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/operator/transformer/filter/transformer.go:43 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(WriterOperator).Write C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/operator/helper/writer.go:54 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/file.(Input).emit C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/operator/input/file/input.go:54 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(Reader).ReadToEnd C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/fileconsumer/internal/reader/reader.go:133 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1 C:/tmp/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/fileconsumer/file.go:160

2024-09-04T16:45:02.929+0200 ERROR reader/reader.go:140 process: %w {"component": "fileconsumer", "path": "testdata\it\functional\logs\otel-collector.log", "error": "regex pattern does not match"} github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(Reader).ReadToEnd C:/Users/sjalaleddine/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/fileconsumer/internal/reader/reader.go:140 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(Manager).consume.func1 C:/Users/sjalaleddine/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/fileconsumer/file.go:160

Quiet

2024-09-04T16:45:02.929+0200 ERROR reader/reader.go:140 process: %w {"component": "fileconsumer", "path": "testdata\it\functional\logs\otel-collector.log", "error": "regex pattern does not match"} github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(Reader).ReadToEnd C:/Users/sjalaleddine/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/fileconsumer/internal/reader/reader.go:140 github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(Manager).consume.func1 C:/Users/sjalaleddine/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/fileconsumer/file.go:160

Collector version

0.107.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

# Sample Operator config with silent errors

- type: regex_parser
  parse_from: body.message
  on_error: drop_quiet
  regex: '^Regex that can sometimes fail$'

Log output

2024-09-04T16:45:02.929+0200 ERROR reader/reader.go:140 process: %w {"component": "fileconsumer", "path": "testdata\it\functional\logs\otel-collector.log", "error": "regex pattern does not match"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd
C:/Users/sjalaleddine/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/fileconsumer/internal/reader/reader.go:140
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
C:/Users/sjalaleddine/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.107.0/fileconsumer/file.go:160

Additional context

No response

github-actions[bot] commented 1 week ago

Pinging code owners: