childe / gohangout

使用 golang 模仿的 Logstash。用于消费 Kafka 数据,处理后写入 ES、Clickhouse 等。
MIT License
1.01k stars 234 forks source link

hangout 不定时crash #247

Closed zcola closed 2 weeks ago

zcola commented 1 month ago

按以下配置测试一下并给出结果 版本1.8.1

- Elasticsearch:
    es_version: 7
    bulk_actions: 36000
    bulk_size: 25
    concurrent: 4
    flush_interval: 60
    hosts:
            - 'http://7.32.146.88:9200'
            - 'http://10.90.142.142:9200'
    index: dasdaogtail_write
    index_type: _doc
    retry_response_code:
    - 429
    sniff:
      match: EQ($.attributes.zone,"hot")
      refresh_interval: 60
I0711 17:35:44.065037       1 bulk_http.go:153] bulk 28119 docs with execution_id 1711
I0711 17:35:44.065099       1 bulk_http.go:163] try to bulk with host (http://7.38.228.72:9200)
I0711 17:35:44.065544       1 brokers.go:133] got 36 brokers
panic: interface conversion: interface {} is nil, not map[string]interface {}
goroutine 170 [running]:
github.com/childe/gohangout/output.filterNodesIPList(0xc0b774a000, {0xc0001af3a0, 0xc0aa392400})
#011/mnt/c/work/gohangout_1.8.1/gohangout/output/elasticsearch_output.go:467 +0x54f
github.com/childe/gohangout/output.sniffNodesFromOneHost({0xc0001c7518, 0xc0003e0a50}, {0xc0001af3a0, 0x1b})
#011/mnt/c/work/gohangout_1.8.1/gohangout/output/elasticsearch_output.go:455 +0x27f
github.com/childe/gohangout/output.sniffNodes(0xc0002d4180)
#011/mnt/c/work/gohangout_1.8.1/gohangout/output/elasticsearch_output.go:419 +0x245
github.com/childe/gohangout/output.newElasticsearchOutput.func3()
#011/mnt/c/work/gohangout_1.8.1/gohangout/output/elasticsearch_output.go:369 +0xb2
created by github.com/childe/gohangout/output.newElasticsearchOutput
#011/mnt/c/work/gohangout_1.8.1/gohangout/output/elasticsearch_output.go:367 +0x13e5
zcola commented 1 month ago

调整 refresh_interval: 3600 应该可以减缓crash 概率,es版本 8.8.2

childe commented 1 month ago

我看代码里面修复过这个panic问题。要么升一下版本到最新的看?