search

elodina / go_kafka_client

Apache Kafka Client Library for Go
http://www.elodina.net
Apache License 2.0
275 stars 74 forks source link

Failed awaiting on state barrier #201

Open lwmonster opened 7 years ago

lwmonster commented 7 years ago

My program has running for several days, and then it crushed. Checking the log, I found this error: 2016-12-28/02:31:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership 2016-12-28/02:31:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership 2016-12-28/02:31:47 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:31:47 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:32:17 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703]

This error repeated 6 times and panic.

I did not run other consumers, and it just comes out.

I have checked #139 and it doesn't work for me, any clue?

lwmonster commented 7 years ago

Full log:

2016-12-28/02:30:47 [INFO] [zk] Received zkEvent Type: EventNodeChildrenChanged State: Unknown Path: /consumers/queryengine/ids 2016-12-28/02:30:47 [INFO] [zk] Trying to renew watcher for consumer registry 2016-12-28/02:30:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] rebalance triggered for 2f33e845-4de4-e780-956d-40e9950b01da 2016-12-28/02:30:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership 2016-12-28/02:30:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership 2016-12-28/02:30:47 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:30:47 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:31:17 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703] 2016-12-28/02:31:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership 2016-12-28/02:31:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership 2016-12-28/02:31:17 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:31:17 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 GC forced gc 1115 @47606.472s 0%: 1.5+10+0.61+53+5.6 ms clock, 3.1+10+0+0/25/84+11 ms cpu, 244->244->226 MB, 454 MB goal, 2 P 2016-12-28/02:31:47 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703] 2016-12-28/02:31:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership 2016-12-28/02:31:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership 2016-12-28/02:31:47 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:31:47 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:32:17 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703] 2016-12-28/02:32:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership 2016-12-28/02:32:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership 2016-12-28/02:32:17 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:32:17 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:32:47 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703] 2016-12-28/02:32:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership 2016-12-28/02:32:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership 2016-12-28/02:32:47 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:32:47 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:33:08 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Starting coordinator cleanup of API reqeusts for queryengine scvg317: inuse: 261, idle: 599, sys: 860, released: 479, consumed: 380 (MB) 2016-12-28/02:33:17 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703] 2016-12-28/02:33:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership 2016-12-28/02:33:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership 2016-12-28/02:33:17 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:33:17 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:33:47 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703] panic: Could not reach consensus on state barrier. goroutine 23971565 [running]: github.com/stealthly/go_kafka_client.(Consumer).rebalance.func1() /home/liangwei/gowork/src/github.com/stealthly/go_kafka_client/consumer.go:776 +0x1038 github.com/stealthly/go_kafka_client.inLock(0xc82065c028, 0xc84dcadf68) /home/liangwei/gowork/src/github.com/stealthly/go_kafka_client/utils.go:44 +0x52 github.com/stealthly/go_kafka_client.(Consumer).rebalance(0xc82065c000) /home/liangwei/gowork/src/github.com/stealthly/go_kafka_client/consumer.go:801 +0x5d created by github.com/stealthly/go_kafka_client.(*Consumer).subscribeForChanges.func1 /home/liangwei/gowork/src/github.com/stealthly/go_kafka_client/consumer.go:704 +0x489`