search

orb-community / orb

Orb is a dynamic network observability platform with agent fleet orchestration and data pipelines with OpenTelemetry
https://orb.community
Mozilla Public License 2.0
575 stars 29 forks source link

panic: runtime error: invalid memory address or nil pointer dereference #3003

Open maluueu opened 6 months ago

maluueu commented 6 months ago

Describe the bug OpenTelemetry agent crashes

To Reproduce Steps to reproduce the behavior:

  1. Configure agent, policy and sink
  2. See error in log

Expected behavior Not crash

Desktop (please complete the following information):

Additional context

orb-agent-1  | Configuration file not provided, default configuration file location is /opt/orb/agent.yaml
orb-agent-1  | Setting default_pcap as visor tap
orb-agent-1  | Running with default config file and pktvisor built-in configuration
orb-agent-1  | Starting orb-agent : /usr/local/bin/orb-agent with args 3
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.390Z","caller":"agent/main.go:81","msg":"backends loaded","backends":{"pktvisor":{"api_host":"localhost","api_port":"10853","binary":"/usr/local/sbin/pktvisord","config_file":"/tmp/orb-agent-pktvisor-conf.kgUO9A"}}}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.391Z","caller":"agent/agent.go:88","msg":"using local config db","filename":"./orb-agent.db"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.392Z","caller":"agent/agent.go:164","msg":"agent started","version":"0.29.0-763b908","routine":"agentRoutine"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.393Z","caller":"cloud_config/cloud_config.go:173","msg":"using explicitly specified cloud configuration","address":"tls://orb.live:8883","id":"a588a8ba-ddd0-4375-b06b-6cd19a382ef4"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.851Z","caller":"agent/agent.go:107","msg":"registered backends","values":["pktvisor","otel","diode"]}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.851Z","caller":"agent/agent.go:108","msg":"requested backends","values":{"pktvisor":{"api_host":"localhost","api_port":"10853","binary":"/usr/local/sbin/pktvisord","config_file":"/tmp/orb-agent-pktvisor-conf.kgUO9A"}}}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.851Z","caller":"pktvisor/pktvisor.go:319","msg":"OpenTelemetry enabled"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.852Z","caller":"pktvisor/pktvisor.go:336","msg":"configured otel receiver host","host":"localhost","port":41841}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.852Z","caller":"pktvisor/pktvisor.go:189","msg":"pktvisor startup","arguments":["--admin-api","-l","localhost","-p","10853","--config","/tmp/orb-agent-pktvisor-conf.kgUO9A","--otel","--otel-host","localhost","--otel-port","41841","--cp-token","dbe207259012e28860ea5a179b9e95b183d318f4cdc7f60a10fe11f95511c299","--cp-url","https://pktvisor.bugsplat.com/post/bp/crash/crashpad.php","--cp-path","/usr/local/sbin/crashpad_handler","--default-geo-city","/geo-db/city.mmdb","--default-geo-asn","/geo-db/asn.mmdb","--default-service-registry","/iana/custom-iana.csv","--cp-custom","a588a8ba-ddd0-4375-b06b-6cd19a382ef4"]}
orb-agent-1  | {"level":"error","ts":"2024-02-15T16:11:42.854Z","caller":"pktvisor/utils.go:42","msg":"received error from payload","error":"Get \"http://localhost:10853/api/v1/metrics/app\": dial tcp [::1]:10853: connect: connection refused"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.854Z","caller":"agent/comms.go:102","msg":"waiting until a backend is in running state","backend":"pktvisor","current state":"process running, REST API unavailable","wait time":"1s"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.909Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.909] [visor] [info] pktvisor 4.4.0 starting up"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.910Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load input stream plugin: dnstap version 1.0 interface visor.module.input/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.910Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load input stream plugin: flow version 1.0 interface visor.module.input/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.910Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load input stream plugin: sflow version 1.0 interface visor.module.input/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.911Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load input stream plugin: mock version 1.0 interface visor.module.input/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.911Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load input stream plugin: netprobe version 1.0 interface visor.module.input/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.911Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load input stream plugin: pcap version 1.0 interface visor.module.input/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.911Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load stream handler plugin: bgp version 1.0 interface visor.module.handler/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.911Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load stream handler plugin: dhcp version 1.0 interface visor.module.handler/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.912Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load stream handler plugin: dns version 1.0 interface visor.module.handler/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.912Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load stream handler plugin: dns version 2.0 interface visor.module.handler/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.912Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load stream handler plugin: flow version 1.0 interface visor.module.handler/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.912Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load stream handler plugin: input_resources version 1.0 interface visor.module.handler/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.912Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load stream handler plugin: net version 1.0 interface visor.module.handler/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.912Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load stream handler plugin: netprobe version 1.0 interface visor.module.handler/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.913Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load stream handler plugin: net version 2.0 interface visor.module.handler/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.913Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Load stream handler plugin: pcap version 1.0 interface visor.module.handler/1.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.913Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Initialize server control plane"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.913Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.910] [visor] [info] Registering DELETE /api/v1/server"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.914Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.913] [visor] [info] Registering GET /api/v1/metrics/app"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.914Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.913] [visor] [info] Registering GET /api/v1/metrics/rates"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.914Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.913] [visor] [info] Registering GET /api/v1/metrics/bucket/(\\d+)"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.914Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.914] [visor] [info] Registering GET /api/v1/metrics/window/(\\d+)"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.915Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.914] [visor] [info] enabling prometheus metrics for \"default\" policy on: /metrics"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.915Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.914] [visor] [info] Registering GET /metrics"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.915Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.914] [visor] [info] Registering GET /api/v1/taps"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.915Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.914] [visor] [info] Registering GET /api/v1/taps/([a-zA-Z_][a-zA-Z0-9_-]*)"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.915Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.914] [visor] [info] Registering POST /api/v1/taps"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.915Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.914] [visor] [info] Registering DELETE /api/v1/taps/([a-zA-Z_][a-zA-Z0-9_-]*)"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.918Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.916] [visor] [info] Registering GET /api/v1/policies"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.918Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.916] [visor] [info] Registering POST /api/v1/policies"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.918Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.916] [visor] [info] Registering GET /api/v1/policies/([a-zA-Z_][a-zA-Z0-9_-]*)"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.918Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.916] [visor] [info] Registering DELETE /api/v1/policies/([a-zA-Z_][a-zA-Z0-9_-]*)"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.918Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.916] [visor] [info] Registering GET /api/v1/policies/([a-zA-Z_][a-zA-Z0-9_-]*)/metrics/(window|bucket)/(\\d+)"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.918Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.916] [visor] [info] Registering GET /api/v1/policies/([a-zA-Z_][a-zA-Z0-9_-]*)/metrics/prometheus"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.924Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.924] [visor] [info] tap [default_pcap]: parsing"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.925Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.925] [visor] [info] tap [default_pcap]: loaded, type pcap"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.926Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.926] [visor] [info] sending anonymous usage metrics (once/day, use --no-track to disable): 0.4.4.pktvisord.metrics.pktvisor.dev."}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:42.987Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:42.987] [visor] [info] web server listening on localhost:10853"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:43.853Z","caller":"pktvisor/pktvisor.go:241","msg":"pktvisor process started","pid":19}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:43.855Z","caller":"pktvisor/pktvisor.go:248","msg":"pktvisor readiness ok, got version ","pktvisor_version":"4.4.0"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:43.855Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:43.854] [visor] [info] REQUEST: GET /api/v1/metrics/app 200"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:43.855Z","caller":"pktvisor/scrape.go:193","msg":"waiting until mqtt client is connected try 1 from 10"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:43.855Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:43.855] [visor] [info] REQUEST: GET /api/v1/metrics/app 200"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.002Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:44.002] [visor] [info] REQUEST: GET /api/v1/metrics/app 200"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.003Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:44.003] [visor] [info] REQUEST: GET /api/v1/taps 200"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.044Z","caller":"agent/rpc_to.go:51","msg":"sending capabilities","value":"{\"schema_version\":\"1.0\",\"orb_agent\":{\"version\":\"0.29.0-763b908\"},\"agent_tags\":null,\"backends\":{\"pktvisor\":{\"version\":\"4.4.0\",\"data\":{\"taps\":{\"default_pcap\":{\"config\":{\"iface\":\"auto\"},\"input_type\":\"pcap\",\"interface\":\"visor.module.input/1.0\",\"tags\":null}}}}}}"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.357Z","caller":"agent/comms.go:236","msg":"completed RPC subscription to group","group_id":"326175d2-ab27-4a74-b771-e23ed5febfa4","group_name":"Router_Group","topic":"channels/538f83aa-9e04-4f07-8491-4185424933da/messages/fromcore"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.459Z","caller":"agent/rpc_to.go:93","msg":"execution period of the re-request of retryGroupMembership","waiting period":0.211574464}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.479Z","caller":"policyMgr/manager.go:57","msg":"managing agent policy from core","action":"manage","name":"Router_Policy","dataset":"4a3f14bb-6174-4c65-8f64-314b518ed606","backend":"pktvisor","id":"05265dfc-a2a3-4a0a-8448-d6fcaf4c2fe6","version":0}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.480Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:44.480] [visor] [info] policy [Router_Policy]: parsing"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.480Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:44.480] [visor] [info] policy [Router_Policy]: creating input stream: default_pcap-c25fbaf2aced4a46"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.481Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:44.481] [visor] [info] policy [Router_Policy]: instantiating Handler default_dns of type dns"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.492Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:44.492] [visor] [info] policy [Router_Policy]: instantiating Handler default_net of type net"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.500Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:44.500] [visor] [info] policy [default_pcap-c25fbaf2aced4a46-resources]: starting"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.536Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:44.536] [visor] [info] policy [Router_Policy]: starting"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.537Z","caller":"policyMgr/manager.go:210","msg":"policy applied successfully","policy_id":"05265dfc-a2a3-4a0a-8448-d6fcaf4c2fe6","policy_name":"Router_Policy"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.537Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:44.537] [visor] [info] REQUEST: GET /api/v1/metrics/app 200"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.538Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:11:44.538] [visor] [info] REQUEST: POST /api/v1/policies 201"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.639Z","caller":"agent/rpc_to.go:153","msg":"execution period of the re-request of retryAgentPolicy","period":0.179541783}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.857Z","caller":"pktvisor/scrape.go:94","msg":"Started receiver for OTLP in orb-agent","host":"localhost","port":41841}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:11:44.857Z","caller":"otlpreceiver@v0.91.0/otlp.go:101","msg":"Starting HTTP server","endpoint":"localhost:41841"}
orb-agent-1  | {"level":"info","ts":"2024-02-15T16:12:33.856Z","caller":"pktvisor/pktvisor.go:212","msg":"pktvisor stdout","log":"[2024-02-15 16:12:33.856] [visor] [info] REQUEST: GET /api/v1/metrics/app 200"}
orb-agent-1  | panic: runtime error: invalid memory address or nil pointer dereference
orb-agent-1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa6e65c]
orb-agent-1  |
orb-agent-1  | goroutine 69 [running]:
orb-agent-1  | go.opentelemetry.io/collector/pdata/pcommon.Value.Type(...)
orb-agent-1  |  /go/pkg/mod/go.opentelemetry.io/collector/pdata@v1.0.0/pcommon/value.go:179
orb-agent-1  | go.opentelemetry.io/collector/pdata/pcommon.Value.AsString({0x0?, 0xc0002050fc?})
orb-agent-1  |  /go/pkg/mod/go.opentelemetry.io/collector/pdata@v1.0.0/pcommon/value.go:374 +0x1c
orb-agent-1  | github.com/orb-community/orb/agent/otel/otlpmqttexporter.(*baseExporter).pushMetrics(0xc000153420, {0x132d5b0?, 0xc0002ffef0}, {0xc0002b7ba8?, 0xc0002050fc?})
orb-agent-1  |  /go/src/github.com/orbcommunity/orb/agent/otel/otlpmqttexporter/otlp.go:151 +0x1fb
orb-agent-1  | go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export(0xc000066800?, {0x132d5b0?, 0xc0002ffef0?})
orb-agent-1  |  /go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.91.0/exporterhelper/metrics.go:58 +0x31
orb-agent-1  | go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send(0xc000290800?, {0x132d5b0?, 0xc0002ffef0?}, {0x1326f10?, 0xc0000121f8?})
orb-agent-1  |  /go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.91.0/exporterhelper/timeout_sender.go:38 +0x98
orb-agent-1  | go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send(0xc0000a8b40, {0x132d5b0, 0xc0002ffef0}, {0x1326f10, 0xc0000121f8})
orb-agent-1  |  /go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.91.0/exporterhelper/retry_sender.go:119 +0x41f
orb-agent-1  | go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send(0xc0002febd0, {0x132df60?, 0xc00016b730?}, {0x1326f10?, 0xc0000121f8?})
orb-agent-1  |  /go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.91.0/exporterhelper/metrics.go:170 +0x7e
orb-agent-1  | go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).consume(0xc0000a4a90, {0x132df60, 0xc00016b730}, {0x1326f10, 0xc0000121f8})
orb-agent-1  |  /go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.91.0/exporterhelper/queue_sender.go:115 +0x56
orb-agent-1  | go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue[...]).Consume(...)
orb-agent-1  |  /go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.91.0/exporterhelper/internal/bounded_memory_queue.go:55
orb-agent-1  | go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*QueueConsumers[...]).Start.func1()
orb-agent-1  |  /go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.91.0/exporterhelper/internal/consumers.go:43 +0x79
orb-agent-1  | created by go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*QueueConsumers[...]).Start in goroutine 52
orb-agent-1  |  /go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.91.0/exporterhelper/internal/consumers.go:39 +0x7d
orb-agent-1  | 13 is not running
weyrick commented 6 months ago

Thanks @maluueu. Can you also include the orb-agent image tag you are using, and the policy that is causing the crash?