search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
108 stars 4.93k forks source link

RegisterHostname causes panic if 2 filebeat receivers are configured #41743

Open leehinman opened 6 hours ago

leehinman commented 6 hours ago

If 2 filebeat receivers are configured, and run a panic occurs when the second tries to register it's hostname in the metrics.

https://github.com/elastic/beats/blob/566e63b3372755800e4091d8331afe49edcab630/libbeat/cmd/instance/beat.go#L839

This is because the monitoring package uses a package level global for the monitoring namespace.

https://github.com/elastic/elastic-agent-libs/blob/4a2d1a91a0431a3adaab859025d8b19baeddd5a3/monitoring/namespace.go#L22

We need to figure out how to have unique monitoring per beat receiver

elastic2:elastic-agent-9.0.0-darwin-aarch64% ./elastic-agent otel -c otel.yml
Starting in otel mode
2024-11-21T20:10:43.289-0600    info    service@v0.113.0/service.go:166 Setting up own telemetry...
2024-11-21T20:10:43.289-0600    info    telemetry/metrics.go:70 Serving metrics {"address": "localhost:8888", "metrics level": "Normal"}
2024-11-21T20:10:43.295-0600    info    Home path: [] Config path: [] Data path: [data] Logs path: [logs]   {"kind": "receiver", "name": "filebeatreceiver/2", "data_type": "logs", "service.name": "elastic2", "ecs.version": "1.6.0"}
2024-11-21T20:10:43.307-0600    info    Beat ID: f5d4082a-3dd7-47cb-8e5d-9caff65644e7   {"kind": "receiver", "name": "filebeatreceiver/2", "data_type": "logs", "service.name": "elastic2", "ecs.version": "1.6.0"}
2024-11-21T20:10:43.339-0600    warn    unable to lookup FQDN: could not get FQDN, all methods failed: failed looking up CNAME: lookup elastic2 on 172.16.1.1:53: no such host: failed looking up IP: lookup elastic2: no such host, using hostname = elastic2 as FQDN  {"kind": "receiver", "name": "filebeatreceiver/2", "data_type": "logs", "service.name": "elastic2", "ecs.version": "1.6.0"}
2024-11-21T20:10:43.340-0600    info    publisher   Beat name: elastic2 {"kind": "receiver", "name": "filebeatreceiver/2", "data_type": "logs", "service.name": "elastic2", "ecs.version": "1.6.0"}
2024-11-21T20:10:43.340-0600    error   modules Not loading modules. Module directory not found: module {"kind": "receiver", "name": "filebeatreceiver/2", "data_type": "logs", "service.name": "elastic2", "ecs.version": "1.6.0"}
2024-11-21T20:10:43.340-0600    warn    Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.    {"kind": "receiver", "name": "filebeatreceiver/2", "data_type": "logs", "service.name": "elastic2", "ecs.version": "1.6.0"}
panic: the variable host cannot be re-registered with options

goroutine 1 [running]:
github.com/elastic/elastic-agent-libs/monitoring.panicErr(...)
    github.com/elastic/elastic-agent-libs@v0.17.3/monitoring/registry.go:257
github.com/elastic/elastic-agent-libs/monitoring.setupMetric(0x1400238c880, {0x1086f639e, 0x4}, {0x14003667610?, 0x1, 0x111460108?})
    github.com/elastic/elastic-agent-libs@v0.17.3/monitoring/metrics.go:350 +0xfc
github.com/elastic/elastic-agent-libs/monitoring.NewFunc(0x1400237ff00?, {0x1086f639e, 0x4}, 0x14002382810, {0x14003667610, 0x1, 0x1})
    github.com/elastic/elastic-agent-libs@v0.17.3/monitoring/metrics.go:242 +0x44
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).RegisterHostname(0x14003112870?, 0x8?)
    github.com/elastic/beats/v7@v7.0.0-alpha2.0.20241111113032-56dca4bbda40/libbeat/cmd/instance/beat.go:839 +0x144
github.com/elastic/beats/v7/libbeat/cmd/instance.NewBeatReceiver({{0x108705204, 0x8}, {0x0, 0x0}, {0x0, 0x0}, 0x1, 0x1, {{0x0, 0x0}, ...}, ...}, ...)
    github.com/elastic/beats/v7@v7.0.0-alpha2.0.20241111113032-56dca4bbda40/libbeat/cmd/instance/beat.go:378 +0x9a4
github.com/elastic/beats/v7/x-pack/filebeat/fbreceiver.createReceiver({0x10339adac?, 0x3?}, {{{{0x14001ee0f00, 0x10}}, {0x14001ee0f11, 0x1}}, {0x1400227b180, {0x10b493f00, 0x14001d20240}, {0x10b48ca88, ...}, ...}, ...}, ...)
    github.com/elastic/beats/v7@v7.0.0-alpha2.0.20241111113032-56dca4bbda40/x-pack/filebeat/fbreceiver/factory.go:45 +0x434
go.opentelemetry.io/collector/receiver.CreateLogsFunc.CreateLogs(...)
    go.opentelemetry.io/collector/receiver@v0.113.0/receiver.go:133
go.opentelemetry.io/collector/service/internal/builders.(*ReceiverBuilder).CreateLogs(0x1400237de40, {0x10b4f0b28, 0x1400228a370}, {{{{0x14001ee0f00, 0x10}}, {0x14001ee0f11, 0x1}}, {0x1400227b180, {0x10b493f00, 0x14001d20240}, ...}, ...}, ...)
    go.opentelemetry.io/collector/service@v0.113.0/internal/builders/receiver.go:86 +0x10c
go.opentelemetry.io/collector/service/internal/graph.(*receiverNode).buildComponent(0x1400228bd60, {0x10b4f0b28, 0x1400228a370}, {0x1400227b180, {0x10b493f00, 0x14001d20240}, {0x10b48ca88, 0x1400237b7a0}, 0x1400237f500, 0x1, ...}, ...)
    go.opentelemetry.io/collector/service@v0.113.0/internal/graph/receiver.go:67 +0x6fc
go.opentelemetry.io/collector/service/internal/graph.(*Graph).buildComponents(0x14000171c80, {0x10b4f0b28, 0x1400228a370}, {{0x14002339000, {0x10b493f00, 0x14001d20240}, {0x10b48ca88, 0x1400237b7a0}, 0x1400237f500, 0x1, ...}, ...})
    go.opentelemetry.io/collector/service@v0.113.0/internal/graph/graph.go:291 +0x618
go.opentelemetry.io/collector/service/internal/graph.Build({0x10b4f0b28, 0x1400228a370}, {{0x14002339000, {0x10b493f00, 0x14001d20240}, {0x10b48ca88, 0x1400237b7a0}, 0x1400237f500, 0x1, {0x1400237f0c0, ...}}, ...})
    go.opentelemetry.io/collector/service@v0.113.0/internal/graph/graph.go:90 +0x348
go.opentelemetry.io/collector/service.(*Service).initGraph(_, {_, _}, {{{0x0, 0x0, {0x108700951, 0x7}, 0x0, 0x0, 0x1400206b620, ...}, ...}, ...})
    go.opentelemetry.io/collector/service@v0.113.0/service.go:342 +0x150
go.opentelemetry.io/collector/service.New({_, _}, {{{0x16d8b77a0, 0xf}, {0x10882db41, 0x2c}, {0x14000f83fd8, 0x5}}, 0x14002354fe0, 0x1400232d200, ...}, ...)
    go.opentelemetry.io/collector/service@v0.113.0/service.go:195 +0xca8
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents(0x14002284a20, {0x10b4f0b28, 0x1400228a370})
    go.opentelemetry.io/collector/otelcol@v0.113.0/collector.go:183 +0x414
go.opentelemetry.io/collector/otelcol.(*Collector).Run(0x14002284a20, {0x10b4f0b28, 0x1400228a370})
    go.opentelemetry.io/collector/otelcol@v0.113.0/collector.go:282 +0x40
github.com/elastic/elastic-agent/internal/pkg/otel.Run({0x10b4f0b28, 0x1400228a0f0}, 0x140010ba360, {0x140010b0bc0, 0x1, 0x1})
    github.com/elastic/elastic-agent/internal/pkg/otel/run.go:44 +0x190
github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runCollector({0x10b4f0a80, 0x1102ece88}, {0x140010b0bc0, 0x1, 0x1})
    github.com/elastic/elastic-agent/internal/pkg/agent/cmd/otel.go:81 +0x148
github.com/elastic/elastic-agent/internal/pkg/agent/cmd.newOtelCommandWithArgs.func1(0x140022cc008, {0x1086f631a?, 0x4?, 0x1086f61f6?})
    github.com/elastic/elastic-agent/internal/pkg/agent/cmd/otel.go:32 +0x64
github.com/spf13/cobra.(*Command).execute(0x140022cc008, {0x14002286840, 0x2, 0x2})
    github.com/spf13/cobra@v1.8.1/command.go:985 +0x840
github.com/spf13/cobra.(*Command).ExecuteC(0x1400027cc08)
    github.com/spf13/cobra@v1.8.1/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).Execute(0x140000b2080?)
    github.com/spf13/cobra@v1.8.1/command.go:1041 +0x1c
main.main()
    github.com/elastic/elastic-agent/main.go:32 +0x130
elasticmachine commented 6 hours ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)