search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.91k stars 2.27k forks source link

Unquote operators of filelog receiver error #29325

Closed freshchen closed 7 months ago

freshchen commented 10 months ago

Component(s)

No response

What happened?

Description

helm： 0.73.1 app version：0.88.0 CRI：docker

otel filelog receiver -> loki

I get below log in grafana

{"body":"2023-11-17 21:03:26,830 [appEnv=web-empty.app instance=web-empty-564cd5f7d9-f25dh tid=0d17ab91fb81cb1da5eead2866add518 sid=47a1447df7b57517] [http-nio-8080-exec-5][INFO] c.b.m.cloudnative.web.log.HttpLog.? - Resp httpCode=200 ; body={\"status\":\"UP\"} ; cost=total:2ms\n"}

and i want to remove \n and Escape，then i try Unquote

I didn’t encounter this problem when using promtail before

Expected Result

{"body":"2023-11-17 21:03:26,830 [appEnv=web-empty.app instance=web-empty-564cd5f7d9-f25dh tid=0d17ab91fb81cb1da5eead2866add518 sid=47a1447df7b57517] [http-nio-8080-exec-5][INFO] c.b.m.cloudnative.web.log.HttpLog.? - Resp httpCode=200 ; body={"status":"UP"} ; cost=total:2ms"}

Actual Result

error

Collector version

0.88.0

Environment information

Environment

CRI：docker k8s: 1.24

OpenTelemetry Collector configuration

No response

Log output

2023-11-17T13:19:06.102Z    error   helper/transformer.go:98    Failed to process entry {"kind": "receiver", "name": "filelog", "data_type": "logs", "operator_id": "unquote", "operator_type": "unquote", "error": "invalid syntax", "action": "send", "entry": {"observed_timestamp":"2023-11-17T13:19:06.102295254Z","timestamp":"2023-11-17T13:19:05.729587074Z","body":"2023-11-17T13:19:05.729Z\tinfo\tservice@v0.88.0/telemetry.go:84\tSetting up own telemetry...\n","attributes":{"log.file.path":"/var/log/pods/observability_opentelemetry-collector-agent-7cth6_b16b3e67-213a-430f-844f-471a7f7ec526/opentelemetry-collector/0.log","restart_count":"0","stream":"stderr","time":"2023-11-17T13:19:05.729587074Z","uid":"b16b3e67-213a-430f-844f-471a7f7ec526"},"resource":{"app":"opentelemetry-collector","namespace":"observability","pod":"opentelemetry-collector-agent-7cth6"},"severity":0,"scope_name":""}}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:98
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:90
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/unquote.(*Transformer).Process
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/unquote/unquote.go:63
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:92
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/move.(*Transformer).Process
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/move/move.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:92
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/move.(*Transformer).Process
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/move/move.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:92
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/move.(*Transformer).Process
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/move/move.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:92
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/move.(*Transformer).Process
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/move/move.go:69
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/parser.go:122
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/parser.go:98
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*Parser).Process
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/parser/regex/regex.go:99
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/parser.go:122
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/parser.go:98
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/json.(*Parser).Process
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/parser/json/json.go:62
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/router.(*Transformer).Process
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/router/router.go:130
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/file.(*Input).emit
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/input/file/file.go:52
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/internal/reader/reader.go:106
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:156

Additional context

No response

github-actions[bot] commented 10 months ago

Pinging code owners for receiver/filelog: @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

crobert-1 commented 10 months ago

Hello @freshchen, can you provide the full configuration of the OpenTelemetry Collector that you're using, as well as an example log file that is hitting this error? This would help us give you more direction on what's going wrong here.

djaglowski commented 9 months ago

body={\"status\":\"UP\"}

Is this the part you are trying to unquote? This appears to be a json string, so you would need to use json_parser.

djaglowski commented 7 months ago

Closing as there is no clear action here.

freshchen commented 6 months ago

Thanks for the answer. Later, I found a way to skip this question.