search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.08k stars 4.89k forks source link

[Metricbeat] panic: runtime error: slice bounds out of range [62325:0] #33965

Open henrikno opened 1 year ago

henrikno commented 1 year ago

Getting a panic when starting metricbeat via stack monitoring.

metricbeat.yml

setup.template.overwrite: false

metricbeat.config.modules:
  enabled: true
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 30s

logging.level: info
logging.to_files: true
logging.json: true
logging.files:
  path: /app/logs/beats
  name: metricbeat.log
  keepfiles: 3

queue.mem:
  events: 4096
  flush.min_events: 2048

output.elasticsearch:
  hosts: ["http://localhost:9244"]
  username: x
  password: x
  ssl.verification_mode: full
  bulk_max_size: 250
  allow_older_versions: true

modules.d/elasticsearch.yml:

- module: elasticsearch
  metricsets:
    - ccr
    - enrich
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["https://${HOSTNAME}:9200"]
  username: x
  password: x
  ssl.verification_mode: none
  xpack.enabled: true

Error in stderr:

2022-11-22T10:25:45+0000 panic: runtime error: slice bounds out of range [62325:0]
2022-11-22T10:25:45+0000
2022-11-22T10:25:45+0000 goroutine 103 [running]:
2022-11-22T10:25:45+0000 bytes.(*Buffer).Write(0xc000ca50b0, {0xc000cb8118, 0x1, 0x40})
2022-11-22T10:25:45+0000 /usr/local/go/src/bytes/buffer.go:174 +0x1ea
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/json.writer.write({{0x56044acc8828, 0xc000ca50b0}}, {0xc000cb8118, 0x1, 0x40})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/json/visitor.go:79 +0x64
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/json.(*Visitor).writeByte(0xc000cb80b0, 0x22)
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/json/visitor.go:105 +0x9e
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/json.(*Visitor).OnString(0xc000cb80b0, {0xc009bb6b28, 0x16})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/json/visitor.go:201 +0xc6
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldString(0xc000ca5200, {0x56044a142d20, 0xc011437400})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold_primitives.go:37 +0x75
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc000ca5200, {0x56044a142d20, 0xc011437400})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold.go:105 +0x19b
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldMapInterface(0xc000ca5200, {0x56044a3c4f20, 0xc006fc4030})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold_map.go:55 +0x1e9
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc000ca5200, {0x56044ab4de00, 0xc006fc4030})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold.go:113 +0x354
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldMapInterface(0xc000ca5200, {0x56044a3c4f20, 0xc006fc4000})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold_map.go:55 +0x1e9
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc000ca5200, {0x56044ab4de00, 0xc006fc4000})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold.go:113 +0x354
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldMapInterface(0xc000ca5200, {0x56044a3c4f20, 0xc005f93ec0})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold_map.go:55 +0x1e9
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc000ca5200, {0x56044ab4de00, 0xc005f93ec0})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold.go:113 +0x354
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldMapInlineInterface(0xc000ca5200, {0x56044ab4de00, 0xc000100238, 0x95})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold_map_inline.generated.go:61 +0x185
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.makeFieldInlineFold.func1(0xc000ca5200, {0x56044a68ffa0, 0xc000100220, 0x99})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:317 +0x99
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.makeFieldsFold.func1(0xc000ca5200, {0x56044a68ffa0, 0xc000100220, 0x99})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:198 +0xe6
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.makeStructFold.func1(0xc000ca5200, {0x56044a68ffa0, 0xc000100220, 0x99})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:188 +0xfd
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldAnyReflect(0xc000ca5200, {0x56044a68ffa0, 0xc000100220, 0x99})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:568 +0x10c
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc000ca5200, {0x56044a68ffa0, 0xc000100220})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold.go:116 +0x3c9
2022-11-22T10:25:45+0000 github.com/elastic/go-structform/gotype.(*Iterator).Fold(0xc000ca5200, {0x56044a68ffa0, 0xc000100220})
2022-11-22T10:25:45+0000 /go/pkg/mod/github.com/elastic/go-structform@v0.0.10/gotype/fold.go:93 +0x45
2022-11-22T10:25:45+0000 github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*jsonEncoder).AddRaw(0xc0002995d8, {0x56044a9b6ce0, 0xc001441090})
2022-11-22T10:25:45+0000 /go/src/github.com/elastic/beats/libbeat/esleg/eslegclient/enc.go:118 +0x17b
2022-11-22T10:25:45+0000 github.com/elastic/beats/v7/libbeat/esleg/eslegclient.bulkEncode(0xc000cb0c00, {0x7fafb85985f8, 0xc0002995d8}, {0xc011064000, 0x1f4, 0x200})
2022-11-22T10:25:45+0000 /go/src/github.com/elastic/beats/libbeat/esleg/eslegclient/bulkapi.go:170 +0x10c
2022-11-22T10:25:45+0000 github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Bulk(0xc0004d6600, {0x56044ad18288, 0xc000078070}, {0x0, 0x0}, {0x0, 0x0}, 0x0, {0xc011064000, 0x1f4, ...})
2022-11-22T10:25:45+0000 /go/src/github.com/elastic/beats/libbeat/esleg/eslegclient/bulkapi.go:79 +0x1c8
2022-11-22T10:25:45+0000 github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).publishEvents(0xc0004d6600, {0x56044ad18288, 0xc000078070}, {0xc001440000, 0xfa, 0xfa})
2022-11-22T10:25:45+0000 /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:233 +0x659
2022-11-22T10:25:45+0000 github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).Publish(0xc0004d6600, {0x56044ad18288, 0xc000078070}, {0x56044ad1e528, 0xc0031b8040})
2022-11-22T10:25:45+0000 /go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:189 +0xd4
2022-11-22T10:25:45+0000 github.com/elastic/beats/v7/libbeat/outputs.(*backoffClient).Publish(0xc000ca5350, {0x56044ad18288, 0xc000078070}, {0x56044ad1e528, 0xc0031b8040})
2022-11-22T10:25:45+0000 /go/src/github.com/elastic/beats/libbeat/outputs/backoff.go:61 +0x6e
2022-11-22T10:25:45+0000 github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).publishBatch(0xc00051da40, {0x56044ad1e528, 0xc0031b8040})
2022-11-22T10:25:45+0000 /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client_worker.go:172 +0x202
2022-11-22T10:25:45+0000 github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run(0xc00051da40)
2022-11-22T10:25:45+0000 /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client_worker.go:157 +0x154
2022-11-22T10:25:45+0000 created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker
2022-11-22T10:25:45+0000 /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/client_worker.go:76 +0x3c5

This seems to happen on the master node, which collects more information. Restarting sometimes seems to fix it, so it doesn't look deterministic.

elasticmachine commented 1 year ago

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

botelastic[bot] commented 1 year ago

This issue doesn't have a Team:<team> label.

belimawr commented 1 year ago

Looking by the stack trace it looks to be an issue when encoding some data as JSON from the event. This would also explain why it's not deterministc.

I tried to reproduce it but I could not.

I tried to reproduce it using Elastic-Package (with version 8.5.1) and a local build of Metricbeat with the settings you posted pointing to Elastic-Package's ES. I tried with both, main and v8.5.1.

Which version of Metricbeat did you use @henrikno ?

belimawr commented 1 year ago

If you can reproduce it somehow relaiably, you could try to use log debug to log the events before they're sent to ES, that might help us to find out what is breaking the encoding. You can add this to the logger settings:

logging.level: debug
logging.selectors: ["processors"]
logging.json: true
logging.files:
   rotateeverybytes: 104857600 # about 100MB

Because it's gonna log every event it's good to increase the default file size.

botelastic[bot] commented 7 months ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

mdavis-ciena commented 1 month ago

This issue also seems to be relevant to filebeat.

filebeat.yaml:

logging:
  json:  true
  level: warning

http:
  enabled: true
  host:    "0.0.0.0"
  port:    6001
  pprof:
    enabled: false

filebeat.inputs:
- id: blueplanet-log
  type: filestream
  paths:
  - "/bp2/data/blueplanet.log"
  - "/bp2/data/blueplanet.log.1"

processors:
- drop_fields:
    fields:
    - beat
    - host
    - input
    - log
    - offset
    - prospector
    - source
- rename:
    fields:
    - from: "message"
      to:   "_message"
- decode_json_fields:
    fields:
    - _message
    target: ""
    overwrite_keys: true
- drop_fields:
    fields:
    - "_message"
- drop_event:
    when:
      not:
        equals:
          fwd_to_es: true
- drop_fields:
    fields:
    - "fwd_to_es"
- fingerprint:
    target_field:   "@metadata._id"
    ignore_missing: true
    fields:
    - kubernetes.namespace_name
    - app
    - app_instance
    - kubernetes.container_name
    - pid
    - oplog
    - priority
    - msg
    - timestamp

setup.template:
  name:      "k8slogs"
  pattern:   "logs-*"
  overwrite: true
  json:
    enabled: true
    path:    "${path.config}/logs-template.json"
    name:    "k8slogs"

output.elasticsearch:
  hosts:                ["http://esinternal:9200"]
  index:                "logs-%{+yyyy.MM.dd}"
  allow_older_versions: true

Error:

panic: runtime error: slice bounds out of range [17851:0]

goroutine 119 [running]:
bytes.(*Buffer).Write(0xc003ca33e0?, {0xc00080c118?, 0x80?, 0x555c93d4f5e0?})
bytes/buffer.go:172 +0xd6
github.com/elastic/go-structform/json.writer.write(...)
github.com/elastic/go-structform@v0.0.10/json/visitor.go:79
github.com/elastic/go-structform/json.(*Visitor).writeByte(0xc00080c0b0?, 0x60?)
github.com/elastic/go-structform@v0.0.10/json/visitor.go:105 +0x35
github.com/elastic/go-structform/json.(*Visitor).OnString(0xc00080c0b0, {0xc000ed6380, 0x9})
github.com/elastic/go-structform@v0.0.10/json/visitor.go:201 +0x52
github.com/elastic/go-structform/json.(*Visitor).OnKey(0xc00080c0b0, {0xc000ed6380?, 0x9?})
github.com/elastic/go-structform@v0.0.10/json/visitor.go:145 +0x6a
github.com/elastic/go-structform/gotype.foldMapInlineInterface(0xc000354fc0, {0x555c923e4540?, 0xc0023db518?, 0xc0007c4df0?})
github.com/elastic/go-structform@v0.0.10/gotype/fold_map_inline.generated.go:58 +0x102
github.com/elastic/go-structform/gotype.makeFieldInlineFold.func1(0xc000354fc0?, {0x555c921b4520?, 0xc0023db500?, 0xc003ca3580?})
github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:317 +0x5d
github.com/elastic/go-structform/gotype.makeFieldsFold.func1(0xc0023db500?, {0x555c921b4520?, 0xc0023db500?, 0xc003ca35a8?})
github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:198 +0x89
github.com/elastic/go-structform/gotype.makeStructFold.func1(0xc000354fc0, {0x555c921b4520?, 0xc0023db500?, 0x555c8f18cbf4?})
github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:188 +0x83
github.com/elastic/go-structform/gotype.foldAnyReflect(0x555c921b4520?, {0x555c921b4520?, 0xc0023db500?, 0x555c8f195905?})
github.com/elastic/go-structform@v0.0.10/gotype/fold_reflect.go:568 +0x9d
github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc000354fc0, {0x555c921b4520?, 0xc0023db500})
github.com/elastic/go-structform@v0.0.10/gotype/fold.go:116 +0x27f
github.com/elastic/go-structform/gotype.(*Iterator).Fold(...)
github.com/elastic/go-structform@v0.0.10/gotype/fold.go:93
github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*jsonEncoder).AddRaw(0xc0003763c0, {0x555c9231f3c0?, 0xc0017be550?})
github.com/elastic/beats/v7/libbeat/esleg/eslegclient/enc.go:118 +0xc5
github.com/elastic/beats/v7/libbeat/esleg/eslegclient.bulkEncode(0xc0007c4d80, {0x7fe9144c5100, 0xc0003763c0}, {0xc0023bd800?, 0x64, 0x555c9172ff6a?})
github.com/elastic/beats/v7/libbeat/esleg/eslegclient/bulkapi.go:170 +0xb7
github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Bulk(0xc000002c00, {0x555c924ff4f8?, 0xc00005e068}, {0x0, 0x0}, {0x0, 0x0}, 0x0, {0xc0023bd800, 0x64, ...})
github.com/elastic/beats/v7/libbeat/esleg/eslegclient/bulkapi.go:79 +0x116
github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).publishEvents(0xc000002c00, {0x555c924ff4f8?, 0xc00005e068?}, {0xc0017be000, 0x32, 0x32})
github.com/elastic/beats/v7/libbeat/outputs/elasticsearch/client.go:248 +0x3d9
github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).Publish(0xc000002c00, {0x555c924ff4f8, 0xc00005e068}, {0x555c92508e40, 0xc0024d1e80})
github.com/elastic/beats/v7/libbeat/outputs/elasticsearch/client.go:189 +0x85
github.com/elastic/beats/v7/libbeat/outputs.(*backoffClient).Publish(0xc000355140, {0x555c924ff4f8?, 0xc00005e068?}, {0x555c92508e40?, 0xc0024d1e80?})
github.com/elastic/beats/v7/libbeat/outputs/backoff.go:61 +0x42
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).publishBatch(0xc000466ac0, {0x555c92508e40?, 0xc0024d1e80?})
github.com/elastic/beats/v7/libbeat/publisher/pipeline/client_worker.go:170 +0x213
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run(0xc000466ac0)
github.com/elastic/beats/v7/libbeat/publisher/pipeline/client_worker.go:155 +0xd6
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker
github.com/elastic/beats/v7/libbeat/publisher/pipeline/client_worker.go:74 +0x22f