open-telemetry / opentelemetry-collector

OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
4.36k stars 1.44k forks source link

otel-collector v0.35.0 crashing if receiver otlp protocol http is present #4043

Closed ghost closed 3 years ago

ghost commented 3 years ago

Describe the bug Whenever starting the otel-collector version 0.35.0, if the config file contains the following

receivers:
  otlp:
    protocols:
      http:

The start of the collector will fail with the reported below.

Steps to reproduce Start the otel-collector v 0.35.0 binary using the config file documented below.

What did you expect to see? I expected to have the otel-collector up and running normally, as this configuration is recommended here: https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver/README.md

What did you see instead? The collector failed to start with the following log:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb29c2f]

goroutine 1 [running]:
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.newConfig({0xc00039b2c0, 0x6, 0x100000000000020})
        go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.23.0/config.go:74 +0x40f
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewHandler({0xe0c340, 0xc000070da0}, {0x0, 0x0}, {0xc0001c9550, 0x4, 0x8})
        go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.23.0/handler.go:71 +0x2b9
go.opentelemetry.io/collector/config/confighttp.(*HTTPServerSettings).ToServer(0xc00007a820, {0xe09760, 0xc00015e300}, {0xc00039ad80, {0xe09ee0, 0xc00038a1c0}, {0x0, 0x0}}, {0xc0001c96a8, 0x1, ...})
        go.opentelemetry.io/collector/config/confighttp/confighttp.go:218 +0x47e
go.opentelemetry.io/collector/receiver/otlpreceiver.(*otlpReceiver).startProtocolServers(0xc00038cb40, {0xe226d8, 0xc00000c6c0})
        go.opentelemetry.io/collector/receiver/otlpreceiver/otlp.go:137 +0x37c
go.opentelemetry.io/collector/receiver/otlpreceiver.(*otlpReceiver).Start(0xc00015e240, {0x1, 0xd0c277}, {0xe226d8, 0xc00000c6c0})
        go.opentelemetry.io/collector/receiver/otlpreceiver/otlp.go:166 +0x29
go.opentelemetry.io/collector/internal/sharedcomponent.(*SharedComponent).Start.func1()
        go.opentelemetry.io/collector/internal/sharedcomponent/sharedcomponent.go:74 +0x3e
sync.(*Once).doSlow(0x7f6568d84108, 0x18)
        sync/once.go:68 +0xd2
sync.(*Once).Do(...)
        sync/once.go:59
go.opentelemetry.io/collector/internal/sharedcomponent.(*SharedComponent).Start(0x258a9b3, {0xe22048, 0xc000032088}, {0xe226d8, 0xc00000c6c0})
        go.opentelemetry.io/collector/internal/sharedcomponent/sharedcomponent.go:73 +0xa5
go.opentelemetry.io/collector/service/internal/builder.(*builtReceiver).Start(...)
        go.opentelemetry.io/collector/service/internal/builder/receivers_builder.go:44
go.opentelemetry.io/collector/service/internal/builder.Receivers.StartAll(0xe04410, {0xe22048, 0xc000032088}, {0xe226a0, 0xc00038db80})
        go.opentelemetry.io/collector/service/internal/builder/receivers_builder.go:73 +0x184
go.opentelemetry.io/collector/service.(*service).Start(0xc00038db80, {0xe22048, 0xc000032088})
        go.opentelemetry.io/collector/service/service.go:109 +0x1fe
go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc000156000, {0xe22048, 0xc000032088})
        go.opentelemetry.io/collector/service/collector.go:228 +0x274
go.opentelemetry.io/collector/service.(*Collector).execute(0xc000156000, {0xe22048, 0xc000032088})
        go.opentelemetry.io/collector/service/collector.go:263 +0x4fd
go.opentelemetry.io/collector/service.New.func1(0xc000382780, {0xcffd0b, 0x2, 0x2})
        go.opentelemetry.io/collector/service/collector.go:117 +0x2e
github.com/spf13/cobra.(*Command).execute(0xc000382780, {0xc000030190, 0x2, 0x2})
        github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000382780)
        github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v1.2.1/command.go:902
go.opentelemetry.io/collector/service.(*Collector).Run(...)
        go.opentelemetry.io/collector/service/collector.go:146
main.runInteractive({{0xc0004c1860, 0xc0004c19b0, 0xc0004c1920, 0xc0004c1800}, {{0xcff86c, 0x7}, {0xd0c21b, 0x17}, {0xe04410, 0x14}}, ...})
        go.opentelemetry.io/collector/cmd/otelcol/main.go:51 +0xc8
main.run(...)
        go.opentelemetry.io/collector/cmd/otelcol/main_others.go:23
main.main()
        go.opentelemetry.io/collector/cmd/otelcol/main.go:40 +0x1d8

What version did you use? Version: v0.35.0

What config did you use? Config:

extensions:
  zpages:

receivers:
  otlp:
    protocols:
      http:

processors:

exporters:
  logging:

service:
  extensions: [zpages]
  pipelines:
    traces:
      receivers:
        - otlp
      processors: []
      exporters:
        - logging

Environment OS: "Ubuntu 20.04" Used the binary available here: https://github.com/open-telemetry/opentelemetry-collector/actions/runs/1233371692

bogdandrutu commented 3 years ago

Where did you get this version of the collector?

bogdandrutu commented 3 years ago

@Aneurysm9 this seems to be in otel-go-contrib

Aneurysm9 commented 3 years ago

This appears to result from #4031, which set a MeterProvider on the otelhttp wrapper for the receiver's HTTP server without ensuring that a valid MeterProvider exists. The collector should avoid doing this and the instrumentation configuration should also guard against this.

codeboten commented 3 years ago

I'll address this issue.

ghost commented 3 years ago

@bogdandrutu I got it from the test, but I also tried to compile it locally and the same error occurred. Link to the test: https://github.com/open-telemetry/opentelemetry-collector/actions/runs/1233371692

codeboten commented 3 years ago

@julianocosta89-dt the fix for this has been pushed to main. I've tested the code with the configuration you provided and it appears to be working. Please take a look and re-open if it's not the case.