search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.51k stars 5.55k forks source link

inputs.gnmi with dynamic tagging won't work with more than one CPUs #11006

Closed kb70 closed 2 years ago

kb70 commented 2 years ago

Relevant telegraf.conf

...
[[inputs.gnmi]]
   addresses = ["device1:6030", "device2:6030", "device3:6030", "device4:6030"]
   username = "xxx"
   password = "yyy"

   redial = "15s"
   encoding = "json"

   tagexclude = ["path", "network_instances_network_instance_protocols_protocol_name"]

   ## Build dynamic tag map for Interface descriptions
   [[inputs.gnmi.subscription]]
      name = "if"
      path = "/interfaces/interface/state/description"
      subscription_mode = "on_change"
      tag_only = true

   [[inputs.gnmi.subscription]]
      name = "if_state"
      path = "/interfaces/interface/state"
      subscription_mode = "sample"
      sample_interval = "15s"
      fielddrop = ["description", "mtu", "name", "last_change", "hardware_port", "type"]

Logs from Telegraf

2022-04-20T12:01:41Z I! Using config file: /etc/telegraf/telegraf.conf
2022-04-20T12:01:41Z I! Starting Telegraf 1.22.1
2022-04-20T12:01:41Z I! Loaded inputs: gnmi internal
2022-04-20T12:01:41Z I! Loaded aggregators: 
2022-04-20T12:01:41Z I! Loaded processors: enum (3x) execd regex
2022-04-20T12:01:41Z I! Loaded outputs: file prometheus_client
2022-04-20T12:01:41Z I! Tags enabled: host=xxx
2022-04-20T12:01:41Z I! [agent] Config: Interval:2m0s, Quiet:false, Hostname:"odin02.arista", Flush Interval:10s
2022-04-20T12:01:41Z D! [agent] Initializing plugins
2022-04-20T12:01:41Z D! [agent] Connecting outputs
2022-04-20T12:01:41Z D! [agent] Attempting connection to [outputs.prometheus_client]
2022-04-20T12:01:41Z I! [outputs.prometheus_client] Listening on http://[::]:9273/metrics
2022-04-20T12:01:41Z D! [agent] Successfully connected to outputs.prometheus_client
2022-04-20T12:01:41Z D! [agent] Attempting connection to [outputs.file]
2022-04-20T12:01:41Z D! [agent] Successfully connected to outputs.file
2022-04-20T12:01:41Z I! [processors.execd] Starting process: /etc/telegraf/execd/base64ToFloat []
2022-04-20T12:01:41Z D! [agent] Starting service inputs
2022-04-20T12:01:41Z E! [processors.execd] stderr: "base64ToFloat here: starting up"
2022-04-20T12:01:41Z D! [inputs.gnmi] Connection to gNMI device device1:6030 established
2022-04-20T12:01:41Z D! [inputs.gnmi] Connection to gNMI device device2:6030 established
2022-04-20T12:01:41Z D! [inputs.gnmi] Connection to gNMI device device3:6030 established
2022-04-20T12:01:41Z D! [inputs.gnmi] Connection to gNMI device device4:6030 established
fatal error: concurrent map read and map write

goroutine 79 [running]:
runtime.throw({0x5413263, 0x0})
    /usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0005a5768 sp=0xc0005a5738 pc=0x439091
runtime.mapaccess2_faststr(0x49a6320, 0xc000075699, {0xc001480dc0, 0x19})
    /usr/local/go/src/runtime/map_faststr.go:116 +0x3d4 fp=0xc0005a57d0 sp=0xc0005a5768 pc=0x414e14
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).handleSubscribeResponseUpdate(0xc0003fea80, {0xc000075699, 0x12}, 0xc00064b8a8)
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:327 +0xb27 fp=0xc0005a5d70 sp=0xc0005a57d0 pc=0x17be4e7
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).handleSubscribeResponse(0xc000691570, {0xc000075699, 0xc001069e60}, 0xc00065f7b0)
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:264 +0x4e fp=0xc0005a5dc0 sp=0xc0005a5d70 pc=0x17bd88e
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).subscribeGNMI(0xc0003fea80, {0x5e68590, 0xc001069e60}, {0xc000075699, 0x12}, 0x0, 0x0)
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:256 +0x536 fp=0xc0005a5ee8 sp=0xc0005a5dc0 pc=0x17bd476
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start.func1({0xc000075699, 0x12})
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:158 +0x125 fp=0xc0005a5fc0 sp=0xc0005a5ee8 pc=0x17bc8a5
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start·dwrap·1()
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:167 +0x31 fp=0xc0005a5fe0 sp=0xc0005a5fc0 pc=0x17bc751
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0005a5fe8 sp=0xc0005a5fe0 pc=0x46d8e1
created by github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:155 +0x9d3

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x0)
    /usr/local/go/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0x53ecaae)
    /usr/local/go/src/sync/waitgroup.go:130 +0x71
github.com/influxdata/telegraf/agent.(*Agent).Run(0xc000b8e0d0, {0x5e684e8, 0xc00108c700})
    /go/src/github.com/influxdata/telegraf/agent/agent.go:181 +0x89c
main.runAgent({0x5e684e8, 0xc00108c700}, {0x8d08ae8, 0x0, 0x0}, {0x8d08ae8, 0x0, 0x0})
    /go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:328 +0xc57
main.reloadLoop({0x8d08ae8, 0x0, 0x0}, {0x8d08ae8, 0x0, 0x0})
    /go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:158 +0x28a
main.run(...)
    /go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf_posix.go:8
main.main()
    /go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:501 +0xa9a

goroutine 69 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000228000)
    /go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xb9
created by go.opencensus.io/stats/view.init.0
    /go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x92

goroutine 70 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
    /go/pkg/mod/k8s.io/klog/v2@v2.30.0/klog.go:1181 +0x6a
created by k8s.io/klog/v2.init.0
    /go/pkg/mod/k8s.io/klog/v2@v2.30.0/klog.go:420 +0xfb

goroutine 71 [chan receive]:
github.com/ClickHouse/clickhouse-go.init.0.func1()
    /go/pkg/mod/github.com/!click!house/clickhouse-go@v1.5.4/bootstrap.go:48 +0x2d
created by github.com/ClickHouse/clickhouse-go.init.0
    /go/pkg/mod/github.com/!click!house/clickhouse-go@v1.5.4/bootstrap.go:45 +0x45

goroutine 83 [syscall]:
os/signal.signal_recv()
    /usr/local/go/src/runtime/sigqueue.go:169 +0x98
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:24 +0x19
created by os/signal.Notify.func1.1
    /usr/local/go/src/os/signal/signal.go:151 +0x2c

goroutine 84 [select]:
main.reloadLoop.func1()
    /go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:145 +0x85
created by main.reloadLoop
    /go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:144 +0x245

goroutine 77 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c750a18, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000320980, 0x4, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000320980)
    /usr/local/go/src/internal/poll/fd_unix.go:402 +0x22c
net.(*netFD).accept(0xc000320980)
    /usr/local/go/src/net/fd_unix.go:173 +0x35
net.(*TCPListener).accept(0xc000aaf818)
    /usr/local/go/src/net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).Accept(0xc000aaf818)
    /usr/local/go/src/net/tcpsock.go:262 +0x3d
net/http.(*Server).Serve(0xc000abe1c0, {0x5e0fa78, 0xc000aaf818})
    /usr/local/go/src/net/http/server.go:3002 +0x394
github.com/influxdata/telegraf/plugins/outputs/prometheus_client.(*PrometheusClient).Connect.func1()
    /go/src/github.com/influxdata/telegraf/plugins/outputs/prometheus_client/prometheus_client.go:227 +0x75
created by github.com/influxdata/telegraf/plugins/outputs/prometheus_client.(*PrometheusClient).Connect
    /go/src/github.com/influxdata/telegraf/plugins/outputs/prometheus_client/prometheus_client.go:225 +0x252

goroutine 78 [syscall]:
syscall.Syscall6(0xf7, 0x1, 0x1f, 0xc0004d4d08, 0x1000004, 0x0, 0x0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:43 +0x5
os.(*Process).blockUntilWaitable(0xc000fed950)
    /usr/local/go/src/os/wait_waitid.go:33 +0x9c
os.(*Process).wait(0xc000fed950)
    /usr/local/go/src/os/exec_unix.go:23 +0x28
os.(*Process).Wait(...)
    /usr/local/go/src/os/exec.go:132
os/exec.(*Cmd).Wait(0xc000aba580)
    /usr/local/go/src/os/exec/exec.go:507 +0x54
github.com/influxdata/telegraf/internal/process.(*Process).cmdWait(0xc0004ee630, {0x5e684e8, 0xc000aa8680})
    /go/src/github.com/influxdata/telegraf/internal/process/process.go:178 +0x27c
github.com/influxdata/telegraf/internal/process.(*Process).cmdLoop(0xc0004ee630, {0x5e684e8, 0xc000aa8680})
    /go/src/github.com/influxdata/telegraf/internal/process/process.go:121 +0x5f
github.com/influxdata/telegraf/internal/process.(*Process).Start.func1()
    /go/src/github.com/influxdata/telegraf/internal/process/process.go:65 +0x2e
created by github.com/influxdata/telegraf/internal/process.(*Process).Start
    /go/src/github.com/influxdata/telegraf/internal/process/process.go:64 +0xf3

goroutine 98 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c750760, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc001083440, 0xc00032e000, 0x1)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001083440, {0xc00032e000, 0x400, 0x400})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
os.(*File).read(...)
    /usr/local/go/src/os/file_posix.go:32
os.(*File).Read(0xc000b8e148, {0xc00032e000, 0x7f098c76ffc0, 0xc001083318})
    /usr/local/go/src/os/file.go:119 +0x5e
github.com/influxdata/telegraf/plugins/parsers/influx.(*streamMachine).Next(0xc0004a6048)
    plugins/parsers/influx/machine.go.rl:519 +0x214
github.com/influxdata/telegraf/plugins/parsers/influx.(*StreamParser).Next(0xc00037c3f0)
    /go/src/github.com/influxdata/telegraf/plugins/parsers/influx/parser.go:192 +0x28
github.com/influxdata/telegraf/plugins/processors/execd.(*Execd).cmdReadOutStream(0xc000320780, {0x5db2760, 0xc000b8e148})
    /go/src/github.com/influxdata/telegraf/plugins/processors/execd/execd.go:151 +0x45
github.com/influxdata/telegraf/plugins/processors/execd.(*Execd).cmdReadOut(0x5255420, {0x5db2760, 0xc000b8e148})
    /go/src/github.com/influxdata/telegraf/plugins/processors/execd/execd.go:123 +0x4b
github.com/influxdata/telegraf/internal/process.(*Process).cmdWait.func1()
    /go/src/github.com/influxdata/telegraf/internal/process/process.go:158 +0x48
created by github.com/influxdata/telegraf/internal/process.(*Process).cmdWait
    /go/src/github.com/influxdata/telegraf/internal/process/process.go:157 +0x148

goroutine 99 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c750590, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc001083500, 0xc000048020, 0x1)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001083500, {0xc000048020, 0xfe0, 0xfe0})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
os.(*File).read(...)
    /usr/local/go/src/os/file_posix.go:32
os.(*File).Read(0xc000b8e158, {0xc000048020, 0xc000194050, 0x20})
    /usr/local/go/src/os/file.go:119 +0x5e
bufio.(*Scanner).Scan(0xc0007c6ef0)
    /usr/local/go/src/bufio/scan.go:215 +0x865
github.com/influxdata/telegraf/plugins/processors/execd.(*Execd).cmdReadErr(0xc000320780, {0x5db2760, 0xc000b8e158})
    /go/src/github.com/influxdata/telegraf/plugins/processors/execd/execd.go:177 +0xad
github.com/influxdata/telegraf/internal/process.(*Process).cmdWait.func2()
    /go/src/github.com/influxdata/telegraf/internal/process/process.go:164 +0x48
created by github.com/influxdata/telegraf/internal/process.(*Process).cmdWait
    /go/src/github.com/influxdata/telegraf/internal/process/process.go:163 +0x1b2

goroutine 100 [select]:
github.com/influxdata/telegraf/internal/process.(*Process).cmdWait.func3()
    /go/src/github.com/influxdata/telegraf/internal/process/process.go:170 +0xac
created by github.com/influxdata/telegraf/internal/process.(*Process).cmdWait
    /go/src/github.com/influxdata/telegraf/internal/process/process.go:169 +0x26f

goroutine 80 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000178fc0)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/transport.go:323 +0x85
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/transport.go:338
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00061a370, {0x50d54a0, 0xc0011905f0}, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:961 +0xbb
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:826 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc000375c00, 0xc0006bdd60, 0xc0006bdd30)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:684 +0xd3
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000375c00, {0x50d54a0, 0xc0011905f0})
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:825 +0x11f
github.com/openconfig/gnmi/proto/gnmi.(*gNMISubscribeClient).Recv(0xc0008d4e90)
    /go/pkg/mod/github.com/openconfig/gnmi@v0.0.0-20180912164834-33a1865c3029/proto/gnmi/gnmi.pb.go:2591 +0x4c
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).subscribeGNMI(0xc0003fea80, {0x5e68590, 0xc001069e60}, {0xc0000756e1, 0x12}, 0x0, 0x0)
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:249 +0x565
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start.func1({0xc0000756e1, 0x12})
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:158 +0x125
created by github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:155 +0x9d3

goroutine 81 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000178d80)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/transport.go:323 +0x85
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/transport.go:338
google.golang.org/grpc.(*csAttempt).recvMsg(0xc000372160, {0x50d54a0, 0xc000ffe2d0}, 0x0)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:961 +0xbb
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:826 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc0001fe000, 0xc0005a9d60, 0xc0005a9d30)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:684 +0xd3
google.golang.org/grpc.(*clientStream).RecvMsg(0xc0001fe000, {0x50d54a0, 0xc000ffe2d0})
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:825 +0x11f
github.com/openconfig/gnmi/proto/gnmi.(*gNMISubscribeClient).Recv(0xc0002d84f0)
    /go/pkg/mod/github.com/openconfig/gnmi@v0.0.0-20180912164834-33a1865c3029/proto/gnmi/gnmi.pb.go:2591 +0x4c
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).subscribeGNMI(0xc0003fea80, {0x5e68590, 0xc001069e60}, {0xc000075729, 0x12}, 0x0, 0x0)
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:249 +0x565
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start.func1({0xc000075729, 0x12})
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:158 +0x125
created by github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:155 +0x9d3

goroutine 114 [runnable]:
reflect.Value.Set({0x51db260, 0xc00108cc50, 0x196}, {0x51db260, 0xc000acb620, 0x16})
    /usr/local/go/src/reflect/value.go:1911 +0x172
google.golang.org/protobuf/internal/impl.makeMessageFieldCoder.func3({0xc000ad735a, 0x1e, 0x1e}, {0xc000628ab0}, 0x0, 0x655, {0x6c, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/codec_field.go:218 +0x133
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc0002d7600, {0xc000ad730e, 0x6a, 0x5093f00}, {0xc00108cc40}, 0x0, {0xb0, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/decode.go:139 +0x3ad
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal(0x5093f00, {{}, {0x5f14ce0, 0xc001088d10}, {0xc000ad730e, 0x6a, 0x6a}, 0x0, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/decode.go:62 +0xb6
google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x1, 0x1, 0x0, {0x5df4708, 0xc0001a3b00}}, {0xc000ad730e, 0x6a, 0x6a}, {0x5f14ce0, ...})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/decode.go:93 +0x1d9
google.golang.org/protobuf/proto.UnmarshalOptions.UnmarshalState(...)
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/decode.go:66
google.golang.org/protobuf/internal/impl.consumeMessageSlice({0xc000ad730d, 0xc000276300, 0x83ac69}, {0xc000ad7303}, {0x5f2f930, 0x5093f00}, 0x0, {0x0, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/codec_field.go:558 +0x1ee
google.golang.org/protobuf/internal/impl.makeMessageSliceFieldCoder.func3({0xc000ad730d, 0xc000276300, 0x57e}, {0x66c}, 0x40, 0xc000628990, {0xe0, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/codec_field.go:460 +0x4a
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc0002d74a0, {0xc000ad7302, 0x76, 0x4042d2}, {0xc0005e1378}, 0x0, {0x0, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/decode.go:139 +0x3ad
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal(0x8b14480, {{}, {0x5f14ce0, 0xc001088d00}, {0xc000ad7302, 0x76, 0x76}, 0x0, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/decode.go:62 +0xb6
google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x1, 0x1, 0x0, {0x5df4708, 0xc0001a3b00}}, {0xc000ad7302, 0x76, 0x76}, {0x5f14ce0, ...})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/decode.go:93 +0x1d9
google.golang.org/protobuf/proto.UnmarshalOptions.UnmarshalState(...)
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/decode.go:66
google.golang.org/protobuf/internal/impl.consumeMessage({0xc000ad7301, 0xc00078a598, 0x50d5220}, {0x5db0f40, 0xc001088d00}, 0x34, {0xc8, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/codec_field.go:283 +0x1c5
google.golang.org/protobuf/internal/impl.makeMessageFieldCoder.func3({0xc000ad7301, 0x77, 0x77}, {0x7f098c40f420}, 0x3e, 0x1c, {0x0, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/codec_field.go:220 +0x186
google.golang.org/protobuf/internal/impl.(*MessageInfo).initOneofFieldCoders.func1({0xc000ad7301, 0x77, 0x77}, {0x2}, 0x20, 0xc001500008, {0x0, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/codec_field.go:64 +0x2a5
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc0001e4420, {0xc000ad7300, 0x78, 0xc0003065f0}, {0xc0005e1860}, 0x0, {0xa0, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/decode.go:139 +0x3ad
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal(0xc0003065f0, {{}, {0x5f14ce0, 0xc001088cf0}, {0xc000ad7300, 0x78, 0x78}, 0x0, {0x5df4708, 0xc0001a3b00}})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/impl/decode.go:62 +0xb6
google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x1, 0x1, 0x0, {0x5df4708, 0xc0001a3b00}}, {0xc000ad7300, 0x78, 0x78}, {0x5f14ce0, ...})
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/decode.go:93 +0x1d9
google.golang.org/protobuf/proto.UnmarshalOptions.UnmarshalState(...)
    /go/pkg/mod/google.golang.org/protobuf@v1.27.1/proto/decode.go:66
github.com/golang/protobuf/proto.UnmarshalMerge({0xc000ad7300, 0x78, 0x78}, {0x5e02878, 0xc0003065f0})
    /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:67 +0x14c
github.com/golang/protobuf/proto.Unmarshal({0xc000ad7300, 0x78, 0x78}, {0x5e02878, 0xc0003065f0})
    /go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:58 +0x65
google.golang.org/grpc/encoding/proto.codec.Unmarshal({}, {0xc000ad7300, 0x78, 0x78}, {0x50d54a0, 0xc0003065f0})
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/encoding/proto/proto.go:53 +0x65
google.golang.org/grpc.recv(0xc0004e896c, {0x7f098c68dd98, 0x8d08ae8}, 0x0, {0x0, 0x0}, {0x50d54a0, 0xc0003065f0}, 0x0, 0x0, ...)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/rpc_util.go:760 +0xb5
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0009b80b0, {0x50d54a0, 0xc0003065f0}, 0x4)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:975 +0x2b0
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:826 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc000332100, 0xc0005e1d60, 0xc0005e1d30)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:680 +0x2f6
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000332100, {0x50d54a0, 0xc0003065f0})
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:825 +0x11f
github.com/openconfig/gnmi/proto/gnmi.(*gNMISubscribeClient).Recv(0xc000322b20)
    /go/pkg/mod/github.com/openconfig/gnmi@v0.0.0-20180912164834-33a1865c3029/proto/gnmi/gnmi.pb.go:2591 +0x4c
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).subscribeGNMI(0xc0003fea80, {0x5e68590, 0xc001069e60}, {0xc00118a019, 0x12}, 0x0, 0x0)
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:249 +0x565
github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start.func1({0xc00118a019, 0x12})
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:158 +0x125
created by github.com/influxdata/telegraf/plugins/inputs/gnmi.(*GNMI).Start
    /go/src/github.com/influxdata/telegraf/plugins/inputs/gnmi/gnmi.go:155 +0x9d3

goroutine 115 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runOutputs(0xc000b8e0d0, 0xc000805400)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:773 +0x2b7
github.com/influxdata/telegraf/agent.(*Agent).Run.func1()
    /go/src/github.com/influxdata/telegraf/agent/agent.go:150 +0x5c
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /go/src/github.com/influxdata/telegraf/agent/agent.go:148 +0x525

goroutine 116 [semacquire]:
sync.runtime_Semacquire(0x0)
    /usr/local/go/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0x0)
    /usr/local/go/src/sync/waitgroup.go:130 +0x71
github.com/influxdata/telegraf/agent.(*Agent).runProcessors(0x0, {0xc000aa8780, 0x5, 0x2000})
    /go/src/github.com/influxdata/telegraf/agent/agent.go:572 +0x125
github.com/influxdata/telegraf/agent.(*Agent).Run.func4()
    /go/src/github.com/influxdata/telegraf/agent/agent.go:171 +0x68
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /go/src/github.com/influxdata/telegraf/agent/agent.go:169 +0x785

goroutine 117 [semacquire]:
sync.runtime_Semacquire(0x0)
    /usr/local/go/src/runtime/sema.go:56 +0x25
sync.(*WaitGroup).Wait(0x0)
    /usr/local/go/src/sync/waitgroup.go:130 +0x71
github.com/influxdata/telegraf/agent.(*Agent).runInputs(0xc000b8e0d0, {0x5e684e8, 0xc00108c700}, {0x0, 0x0, 0x8cc64c0}, 0xc0008056e0)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:330 +0x4a6
github.com/influxdata/telegraf/agent.(*Agent).Run.func5()
    /go/src/github.com/influxdata/telegraf/agent/agent.go:178 +0x74
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /go/src/github.com/influxdata/telegraf/agent/agent.go:176 +0x88f

goroutine 118 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000fbc900, {0x5e684e8, 0xc000fbc940}, 0xc000ab0f00)
    /go/src/github.com/influxdata/telegraf/agent/tick.go:86 +0xfe
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
    /go/src/github.com/influxdata/telegraf/agent/tick.go:64 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
    /go/src/github.com/influxdata/telegraf/agent/tick.go:62 +0x185

goroutine 119 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x0, {0x5e684e8, 0xc00108c700}, {0x5edcb20, 0xc0004c69e0}, 0xc000ab0550, {0x5df0478, 0xc000fbc900}, 0x0)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:465 +0x125
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x0)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:326 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
    /go/src/github.com/influxdata/telegraf/agent/agent.go:324 +0xa7

goroutine 120 [select]:
github.com/influxdata/telegraf/agent.(*AlignedTicker).run(0xc000fbc9c0, {0x5e684e8, 0xc000fbca00}, 0xc000ab0ff0)
    /go/src/github.com/influxdata/telegraf/agent/tick.go:86 +0xfe
github.com/influxdata/telegraf/agent.(*AlignedTicker).start.func1()
    /go/src/github.com/influxdata/telegraf/agent/tick.go:64 +0x6a
created by github.com/influxdata/telegraf/agent.(*AlignedTicker).start
    /go/src/github.com/influxdata/telegraf/agent/tick.go:62 +0x185

goroutine 121 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherLoop(0x0, {0x5e684e8, 0xc00108c700}, {0x5edcb20, 0xc0004c6a00}, 0xc000ab05f0, {0x5df0478, 0xc000fbc9c0}, 0x0)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:465 +0x125
github.com/influxdata/telegraf/agent.(*Agent).runInputs.func1(0x0)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:326 +0x7e
created by github.com/influxdata/telegraf/agent.(*Agent).runInputs
    /go/src/github.com/influxdata/telegraf/agent/agent.go:324 +0xa7

goroutine 130 [select]:
github.com/influxdata/telegraf/agent.(*Agent).flushLoop(0xc00078fb90, {0x5e684e8, 0xc000190800}, 0xc000228380, {0x5df04a0, 0xc00078fb90})
    /go/src/github.com/influxdata/telegraf/agent/agent.go:818 +0x1e5
github.com/influxdata/telegraf/agent.(*Agent).runOutputs.func1(0x0)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:769 +0x125
created by github.com/influxdata/telegraf/agent.(*Agent).runOutputs
    /go/src/github.com/influxdata/telegraf/agent/agent.go:763 +0xc5

goroutine 146 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000aaf860)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:561 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
    /go/src/github.com/influxdata/telegraf/agent/agent.go:557 +0x45

goroutine 147 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000aaf8a8)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:561 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
    /go/src/github.com/influxdata/telegraf/agent/agent.go:557 +0x45

goroutine 148 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000aaf8c0)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:561 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
    /go/src/github.com/influxdata/telegraf/agent/agent.go:557 +0x45

goroutine 149 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000aaf8d8)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:561 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
    /go/src/github.com/influxdata/telegraf/agent/agent.go:557 +0x45

goroutine 150 [chan receive]:
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc000aaf8f0)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:561 +0x107
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors
    /go/src/github.com/influxdata/telegraf/agent/agent.go:557 +0x45

goroutine 131 [select]:
github.com/influxdata/telegraf/agent.(*Agent).flushLoop(0xc000b808a0, {0x5e684e8, 0xc000190800}, 0xc000228500, {0x5df04a0, 0xc000b808a0})
    /go/src/github.com/influxdata/telegraf/agent/agent.go:818 +0x1e5
github.com/influxdata/telegraf/agent.(*Agent).runOutputs.func1(0x0)
    /go/src/github.com/influxdata/telegraf/agent/agent.go:769 +0x125
created by github.com/influxdata/telegraf/agent.(*Agent).runOutputs
    /go/src/github.com/influxdata/telegraf/agent/agent.go:763 +0xc5

goroutine 132 [select]:
github.com/influxdata/telegraf/agent.(*RollingTicker).run(0xc000b808a0, {0x5e684e8, 0xc000640040}, 0xc0011903c0)
    /go/src/github.com/influxdata/telegraf/agent/tick.go:261 +0xec
github.com/influxdata/telegraf/agent.(*RollingTicker).start.func1()
    /go/src/github.com/influxdata/telegraf/agent/tick.go:249 +0x6a
created by github.com/influxdata/telegraf/agent.(*RollingTicker).start
    /go/src/github.com/influxdata/telegraf/agent/tick.go:247 +0x165

goroutine 151 [select]:
github.com/influxdata/telegraf/agent.(*RollingTicker).run(0xc00078fb90, {0x5e684e8, 0xc00069e000}, 0xc000b880f0)
    /go/src/github.com/influxdata/telegraf/agent/tick.go:261 +0xec
github.com/influxdata/telegraf/agent.(*RollingTicker).start.func1()
    /go/src/github.com/influxdata/telegraf/agent/tick.go:249 +0x6a
created by github.com/influxdata/telegraf/agent.(*RollingTicker).start
    /go/src/github.com/influxdata/telegraf/agent/tick.go:247 +0x165

goroutine 162 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0007960a0)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/balancer_conn_wrappers.go:77 +0xa7
created by google.golang.org/grpc.newCCBalancerWrapper
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/balancer_conn_wrappers.go:67 +0x246

goroutine 198 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:333 +0x98
created by google.golang.org/grpc.newClientStreamWithParams
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:332 +0xb45

goroutine 161 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c750020, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000338880, 0xc0006ea000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000338880, {0xc0006ea000, 0x8000, 0x8000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000338880, {0xc0006ea000, 0x60100000000, 0x8})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc0006a0030, {0xc0006ea000, 0x43bc20, 0xc00051c1a0})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0000724e0, {0xc00106e580, 0x9, 0xc000643d18})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x5da4680, 0xc0000724e0}, {0xc00106e580, 0x9, 0x9}, 0x9)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc00106e580, 0x9, 0x18}, {0x5da4680, 0xc0000724e0})
    /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00106e540)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0002db860)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:1499 +0x41f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:364 +0x192f

goroutine 122 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000ab1130)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/balancer_conn_wrappers.go:77 +0xa7
created by google.golang.org/grpc.newCCBalancerWrapper
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/balancer_conn_wrappers.go:67 +0x246

goroutine 133 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:333 +0x98
created by google.golang.org/grpc.newClientStreamWithParams
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:332 +0xb45

goroutine 194 [runnable]:
bytes.(*Buffer).Write(0xc00077d8f0, {0xc0009e01c0, 0xc4, 0x1a5})
    /usr/local/go/src/bytes/buffer.go:168 +0xff
google.golang.org/grpc/internal/transport.(*http2Client).handleData(0xc000750000, 0xc0009e4060)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:1079 +0x352
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000750000)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:1534 +0x91c
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:364 +0x192f

goroutine 56 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0003140a0)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/balancer_conn_wrappers.go:77 +0xa7
created by google.golang.org/grpc.newCCBalancerWrapper
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/balancer_conn_wrappers.go:67 +0x246

goroutine 180 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:333 +0x98
created by google.golang.org/grpc.newClientStreamWithParams
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:332 +0xb45

goroutine 152 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000b88230)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/balancer_conn_wrappers.go:77 +0xa7
created by google.golang.org/grpc.newCCBalancerWrapper
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/balancer_conn_wrappers.go:67 +0x246

goroutine 159 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c7502d8, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000379580, 0xc0006d2000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000379580, {0xc0006d2000, 0x8000, 0x8000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000379580, {0xc0006d2000, 0xc00070f400, 0xc000f144b0})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc0006a0028, {0xc0006d2000, 0xc0007b7c78, 0x50e005})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc0000723c0, {0xc00106e4a0, 0x9, 0xc0007b7cb8})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x5da4680, 0xc0000723c0}, {0xc00106e4a0, 0x9, 0x9}, 0x9)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc00106e4a0, 0x9, 0xc000fb44e0}, {0x5da4680, 0xc0000723c0})
    /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00106e460)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0002db680)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:1499 +0x41f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:364 +0x192f

goroutine 210 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000b88640, 0x1)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/controlbuf.go:407 +0x11b
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000072540)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/controlbuf.go:534 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:414 +0x65
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:412 +0x1f85

goroutine 199 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:333 +0x98
created by google.golang.org/grpc.newClientStreamWithParams
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/stream.go:332 +0xb45

goroutine 196 [IO wait]:
internal/poll.runtime_pollWait(0x7f098c750930, 0x72)
    /usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc000379800, 0xc000756000, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000379800, {0xc000756000, 0x8000, 0x8000})
    /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000379800, {0xc000756000, 0x60100000000, 0x8})
    /usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc000736008, {0xc000756000, 0x43bc20, 0xc0007521a0})
    /usr/local/go/src/net/net.go:183 +0x45
bufio.(*Reader).Read(0xc00025b2c0, {0xc0011a8200, 0x9, 0xc000646d18})
    /usr/local/go/src/bufio/bufio.go:227 +0x1b4
io.ReadAtLeast({0x5da4680, 0xc00025b2c0}, {0xc0011a8200, 0x9, 0x9}, 0x9)
    /usr/local/go/src/io/io.go:328 +0x9a
io.ReadFull(...)
    /usr/local/go/src/io/io.go:347
golang.org/x/net/http2.readFrameHeader({0xc0011a8200, 0x9, 0x18}, {0x5da4680, 0xc00025b2c0})
    /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0011a81c0)
    /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:498 +0x95
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0007501e0)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:1499 +0x41f
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:364 +0x192f

goroutine 160 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000b88550, 0x1)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/controlbuf.go:407 +0x11b
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000072420)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/controlbuf.go:534 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:414 +0x65
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:412 +0x1f85

goroutine 195 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000ffe050, 0x1)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/controlbuf.go:407 +0x11b
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00025afc0)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/controlbuf.go:534 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:414 +0x65
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:412 +0x1f85

goroutine 197 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000ffe140, 0x1)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/controlbuf.go:407 +0x11b
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00025b320)
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/controlbuf.go:534 +0x85
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:414 +0x65
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /go/pkg/mod/google.golang.org/grpc@v1.44.0/internal/transport/http2_client.go:412 +0x1f85

System info

Telegraf 1.22.1, Debian Bullseye, Docker 20.10.14

Docker

Docker container is provisioned with ansible, only 'special' setting is memory_limit 512MB

Steps to reproduce

-) run Telegraf docker container with dynamic tagging configuration and more than 4 subscribed addresses.

When there's only one CPU available (1 socket, 1 core) everything work's as expected.

As soon as ther's more than one CPU available (eg. 1 socket, 2 cores) a fatal error arises: fatal error: concurrent map read and map write

Expected behavior

working telegraf container

Actual behavior

fatal error: concurrent map read and map write

Additional info

No response

powersj commented 2 years ago

Hi,

fatal error: concurrent map read and map write

This is probably from this recent addition of gnmi subscriptions. Telegraf may need to either sync the lookup map or put some mutexes in place.

When there's only one CPU available (1 socket, 1 core) everything work's as expected. As soon as ther's more than one CPU available (eg. 1 socket, 2 cores) a fatal error arises:

Out of curiosity, how are you controlling this? You said the only special setting is the memory_limit.

kb70 commented 2 years ago

Hi,

fatal error: concurrent map read and map write

This is probably from this recent addition of gnmi subscriptions. Telegraf may need to either sync the lookup map or put some mutexes in place.

When there's only one CPU available (1 socket, 1 core) everything work's as expected. As soon as ther's more than one CPU available (eg. 1 socket, 2 cores) a fatal error arises:

Out of curiosity, how are you controlling this? You said the only special setting is the memory_limit.

In a test environment, where the docker container runs on a KVM virtualized host, where i can arbitrarily configure the number of sockets and cores.

powersj commented 2 years ago

@bewing as you are the author FYI - curious if you have seen this or if you are interested in looking to resolve this.

bewing commented 2 years ago

Thanks for the heads up -- My primary application was running telegraf directly on the device we were monitoring with gNMI, so I never ran into this.

I believe I just have to add a sync.Mutex to update or read from gnmi.GNMI.lookup ?

bewing commented 2 years ago

@kb70 are you easily able to checkout the PR and see if the change resolves this issue?

Or download one of the build artifacts? https://github.com/influxdata/telegraf/pull/11008#issuecomment-1104241007

kb70 commented 2 years ago

@bewing Thanks for the quick fix. I tested the PR (via PR build artifacts and 'quick & dirty' approach, ie. replacing the telegraf binary in the docker container(Telegraf 1.23.0-dfd54ceb)) and as far as I can tell it works now. Independent of the number of sockets or cores telegraf starts up without errors.

Thanks again for your work on this very valuable feature.

PS.: I'll test it on one of our production monitoring machines, ie. real hardware, later on, but won't be able to report about the outcome before tomorrow afternoon.

powersj commented 2 years ago

Thank you both for the fix and for testing this! I want to get this in our bugfix release Monday, so please do let us know the outcome, but I may merge anyway before then.

kb70 commented 2 years ago

The fixed version runs without problems since about a day now on the virtualized testsystem as well as on bare-metal, both with multiple socket CPUs and cores.

Again thanks for the quick response and fix.

powersj commented 2 years ago

@kb70 thank you for testing!