search

grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.28k stars 174 forks source link

otelcol.exporter.awss3 fails to configure #1687

Open cydergoth opened 3 days ago

cydergoth commented 3 days ago

What's wrong?

alloy, version v1.3.1 (branch: HEAD, revision: e4979b2a2)
  build user:       root@buildkitsandbox
  build date:       2024-08-23T15:42:36Z
  go version:       go1.22.5
  platform:         linux/amd64
  tags:             netgo,builtinassets,promtail_journal_enabled
Error: /etc/alloy/config.river:253:1: Failed to build component: building component: unrecognized debug metric level:

252 |   // OTEL exporter to send the logs to S3
253 |   otelcol.exporter.awss3 "pod_logs_s3_target" {
    |  _^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
254 | |   s3_uploader {
255 | |     region = "us-west-2"
256 | |     s3_bucket = "ops-central-logs"
257 | |     s3_prefix = "/logs/tenant_id"
258 | |     file_prefix = "tenant_id"
259 | |   }
260 | | }
    | |_^

ts=2024-09-14T19:42:40.05325084Z level=error msg="failed to start reporter" err="context canceled"

Steps to reproduce

Create an alloy configuration file which uses has this stanza in it: 

// OTEL exporter to send the logs to S3
otelcol.exporter.awss3 "pod_logs_s3_target" {
  s3_uploader {
    region = "us-west-2"
    s3_bucket = "ops-central-logs"
    s3_prefix = "/logs/tenant_id"
    file_prefix = "tenant_id"
  }
}

Attempt to start Alloy version v1.3.1 with this configuration. Observe the above error

System information

Linux ubuntu-mantic 6.5.0-44-generic #44-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 7 15:10:09 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Software version

Grafana Alloy version v1.3.1

Configuration

// OTEL exporter to send the logs to S3 otelcol.exporter.awss3 "pod_logs_s3_target" { s3_uploader { region = "us-west-2" s3_bucket = "ops-central-logs" s3_prefix = "/txn/tenant_id/logs" file_prefix = "tenant_id_txn" } }

Logs

http://127.0.0.1:8080 ts=2024-09-14T19:42:40.050784621Z level=info "boringcrypto enabled"=false ts=2024-09-14T19:42:40.046988253Z level=info source=/go/pkg/mod/github.com/!kim!machine!gun/automemlimit@v0.6.0/memlimit/memlimit.go:170 msg="memory is not limited, skipping: %v" package=github.com/KimMachineGun/automemlimit/memlimit !BADKEY="memory is not limited" ts=2024-09-14T19:42:40.050907537Z level=info msg="running usage stats reporter" ts=2024-09-14T19:42:40.05090976Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 ts=2024-09-14T19:42:40.050917621Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=otel duration=2.171µs ts=2024-09-14T19:42:40.050921378Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=loki.echo.stdout duration=21.125µs ts=2024-09-14T19:42:40.050923958Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=loki.process.pod_logs_txn_loki duration=414.379µs ts=2024-09-14T19:42:40.050928256Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=logging duration=147.771µs ts=2024-09-14T19:42:40.050961401Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=remotecfg duration=28.124µs ts=2024-09-14T19:42:40.051052494Z level=error msg="failed to evaluate config" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node=otelcol.exporter.awss3.pod_logs_txn_s3_target err="building component: unrecognized debug metric level: " ts=2024-09-14T19:42:40.051063043Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=otelcol.exporter.awss3.pod_logs_txn_s3_target duration=96.565µs ts=2024-09-14T19:42:40.051105554Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=otelcol.receiver.loki.pod_logs_txn_s3_otel duration=35.009µs ts=2024-09-14T19:42:40.051163263Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=loki.process.pod_logs_txn_s3 duration=52.332µs ts=2024-09-14T19:42:40.052129351Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=loki.process.pod_logs_txn duration=922.935µs ts=2024-09-14T19:42:40.052478406Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=loki.source.file.test_files duration=158.7µs ts=2024-09-14T19:42:40.052574419Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=tracing duration=9.763µs ts=2024-09-14T19:42:40.052596036Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=livedebugging duration=14.016µs ts=2024-09-14T19:42:40.052605669Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=labelstore duration=4.972µs ts=2024-09-14T19:42:40.052619786Z level=info msg="applying non-TLS config to HTTP server" service=http ts=2024-09-14T19:42:40.052622336Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=http duration=10.85µs ts=2024-09-14T19:42:40.052629962Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=ui duration=3.609µs ts=2024-09-14T19:42:40.052665464Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 node_id=cluster duration=3.9µs ts=2024-09-14T19:42:40.052672591Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=77a4a2b2762d841fc75c6e513a84c0c6 duration=2.430269ms Error: /etc/alloy/config.river:253:1: Failed to build component: building component: unrecognized debug metric level:

252 | // OTEL exporter to send the logs to S3 253 | otelcol.exporter.awss3 "pod_logs_s3target" { | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 254 | | s3_uploader { 255 | | region = "us-west-2" 256 | | s3_bucket = "ops-central-logs" 257 | | s3_prefix = "/logs/tenant_id" 258 | | file_prefix = "tenantid" 259 | | } 260 | | } | |^

ts=2024-09-14T19:42:40.05325084Z level=error msg="failed to start reporter" err="context canceled" Error: could not perform the initial load successfully

cydergoth commented 3 days ago

Note: Attempting to explicitly set a debug metric level gives : 


// OTEL exporter to send the logs to S3
otelcol.exporter.awss3 "pod_logs_s3_target" {
  debug_metrics {
     level = "none"
     }
  s3_uploader {
    region = "us-west-2"
    s3_bucket = "ops-central-logs"
    s3_prefix = "/logs/tenant_id"
    file_prefix = "tenant_id"
  }
}
panic: reflect.Set: value of type string is not assignable to type otelcol.Level [recovered]
        panic: reflect.Set: value of type string is not assignable to type otelcol.Level [recovered]
        panic: reflect.Set: value of type string is not assignable to type otelcol.Level

goroutine 1 [running]:
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.deferwrap1()
        /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.28.0/trace/span.go:398 +0x25
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc00295f6c0, {0x0, 0x0, 0xe89d480?})
        /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.28.0/trace/span.go:436 +0xa82
panic({0x68eeae0?, 0xc003039c90?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.deferwrap1()
        /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.28.0/trace/span.go:398 +0x25
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc00295fba0, {0x0, 0x0, 0xe89d480?})
        /go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.28.0/trace/span.go:436 +0xa82
panic({0x68eeae0?, 0xc003039c90?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
reflect.Value.assignTo({0x68eeae0?, 0xc003039c80?, 0x10?}, {0x80d2c7a, 0xb}, 0x6c8bc40, 0x0)
        /usr/local/go/src/reflect/value.go:3356 +0x299
reflect.Value.Set({0x6c8bc40?, 0xc0030371d0?, 0x7063e20?}, {0x68eeae0?, 0xc003039c80?, 0x6c8bc40?})
        /usr/local/go/src/reflect/value.go:2325 +0xe6
github.com/grafana/alloy/syntax/internal/value.(*decoder).decode(0xc003080290, {{0x68eeae0?, 0xc003039c70?, 0x414a65?}, 0x78?}, {0x6c8bc40, 0xc0030371d0, 0x198})
        /src/alloy/syntax/internal/value/decode.go:215 +0x1345
github.com/grafana/alloy/syntax/internal/value.Decode({{0x68eeae0?, 0xc003039c70?, 0xc003086d50?}, 0x88?}, {0x6df6480?, 0xc0030371d0?})
        /src/alloy/syntax/internal/value/decode.go:80 +0xf5
github.com/grafana/alloy/syntax/vm.(*structDecoder).decodeAttr(0xc002a0b338, 0xc003076660, {0x74ca6c0?, 0xc0030371c8?, 0x7a092c448758?}, 0xc002a0aaa8)
        /src/alloy/syntax/vm/struct_decoder.go:181 +0x74e
github.com/grafana/alloy/syntax/vm.(*structDecoder).Decode(0xc002a0b338, {0xc003039450, 0x1, 0x0?}, {0x74ca6c0?, 0xc0030371c8?, 0xc0030371c8?})
        /src/alloy/syntax/vm/struct_decoder.go:71 +0x749
github.com/grafana/alloy/syntax/vm.(*Evaluator).evaluateDecode(0xc003039770, 0xc003039b90, 0xc003086d50, {0x94348a0, 0xc003015b80}, {0x6df64e0?, 0xc0030371c8?, 0x2060000000b?})
        /src/alloy/syntax/vm/vm.go:169 +0x37f
github.com/grafana/alloy/syntax/vm.(*Evaluator).evaluateBlockOrBody(0xc003039770, 0xc003039b90, 0xc003086d50, {0x94348a0, 0xc003015b80}, {0x74ca6c0?, 0xc0030371c8?, 0x1?})
        /src/alloy/syntax/vm/vm.go:93 +0x12a
github.com/grafana/alloy/syntax/vm.(*structDecoder).decodeNormalBlock(0xc002a0c278, {0xc003001ee0, 0xd}, 0xc003015b80, {0x7c63600?, 0xc003037110?, 0x10?}, 0xc002a0b9e8)
        /src/alloy/syntax/vm/struct_decoder.go:276 +0x894
github.com/grafana/alloy/syntax/vm.(*structDecoder).decodeBlock(0xc002a0c278, 0xc003015b80, {0x7c63600?, 0xc003037110?, 0x7a092c448758?}, 0xc002a0b9e8)
        /src/alloy/syntax/vm/struct_decoder.go:195 +0x152
github.com/grafana/alloy/syntax/vm.(*structDecoder).Decode(0xc002a0c278, {0xc0030790c0, 0x2, 0x0?}, {0x7c63600?, 0xc003037110?, 0xc002a0c288?})
        /src/alloy/syntax/vm/struct_decoder.go:77 +0x78d
github.com/grafana/alloy/syntax/vm.(*Evaluator).evaluateDecode(0xc003039770, 0xc003039b90, 0xc003086d50, {0x94325e0, 0xc0030767f8}, {0x7552be0?, 0xc003037110?, 0xe8a2e28?})
        /src/alloy/syntax/vm/vm.go:169 +0x37f
github.com/grafana/alloy/syntax/vm.(*Evaluator).evaluateBlockOrBody(0xc003039770, 0xc003039b90, 0xc003086d50, {0x94325e0, 0xc0030767f8}, {0x7552be0?, 0xc003037110?, 0x7552be0?})
        /src/alloy/syntax/vm/vm.go:93 +0x12a
github.com/grafana/alloy/syntax/vm.(*Evaluator).Evaluate(0xc003039770, 0xc003039b90, {0x7552be0, 0xc003037110})
        /src/alloy/syntax/vm/vm.go:58 +0x2ce
github.com/grafana/alloy/internal/runtime/internal/controller.(*BuiltinComponentNode).evaluate(0xc0021afb08, 0xc003039b90)
        /src/alloy/internal/runtime/internal/controller/node_builtin_component.go:265 +0x177
github.com/grafana/alloy/internal/runtime/internal/controller.(*BuiltinComponentNode).Evaluate(0xc0021afb08, 0x7332140?)
        /src/alloy/internal/runtime/internal/controller/node_builtin_component.go:248 +0x1c
github.com/grafana/alloy/internal/runtime/internal/controller.(*Loader).evaluate(0xc00295f520, {0x9432420, 0xc00307e5a0}, {0x94b2708, 0xc0021afb08})
        /src/alloy/internal/runtime/internal/controller/loader.go:831 +0x49
github.com/grafana/alloy/internal/runtime/internal/controller.(*Loader).Apply.func2({0x7a092c44d790, 0xc0021afb08})
        /src/alloy/internal/runtime/internal/controller/loader.go:201 +0x1085
github.com/grafana/alloy/internal/runtime/internal/dag.WalkTopological(0xc003079100, {0xc003011600, 0x9, 0xdd03ff0?}, 0xc002a0d240)
        /src/alloy/internal/runtime/internal/dag/walk.go:83 +0x222
github.com/grafana/alloy/internal/runtime/internal/controller.(*Loader).Apply(0xc00295f520, {0x0, {0xc00307a680, 0x7, 0x8}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...})
        /src/alloy/internal/runtime/internal/controller/loader.go:184 +0xb2d
github.com/grafana/alloy/internal/runtime.(*Runtime).loadSource(0xc003010f00, 0xc0021a1e30, 0x0, 0x0)
        /src/alloy/internal/runtime/alloy.go:316 +0x1b8
github.com/grafana/alloy/internal/runtime.(*Runtime).LoadSource(...)
        /src/alloy/internal/runtime/alloy.go:300
github.com/grafana/alloy/internal/alloycli.(*alloyRun).Run.func4()
        /src/alloy/internal/alloycli/cmd_run.go:324 +0x28a
github.com/grafana/alloy/internal/alloycli.(*alloyRun).Run(0xc00013bb00, {0x7ffce0b32f49, 0x17})
        /src/alloy/internal/alloycli/cmd_run.go:357 +0x1335
github.com/grafana/alloy/internal/alloycli.runCommand.func1(0xc00013bc00?, {0xc003006b40?, 0x4?, 0x80b6267?})
        /src/alloy/internal/alloycli/cmd_run.go:103 +0x2b
github.com/spf13/cobra.(*Command).execute(0xc00300c608, {0xc003006ab0, 0x3, 0x3})
        /go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xaca
github.com/spf13/cobra.(*Command).ExecuteC(0xc00180b808)
        /go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/grafana/alloy/internal/alloycli.Run()
        /src/alloy/internal/alloycli/alloycli.go:33 +0x2f8
main.main()
        /src/alloy/main.go:35 +0xf
cydergoth commented 3 days ago

Workaround, adding this to the configuration stanza works:

        debug_metrics {
           disable_high_cardinality_metrics = true
           }

Looking at the tests for the awss3 exporter all the tests include this section, which is probably why the issue wasn't caught