search

btccom / btcpool-ABANDONED

backend of pool.btc.com
https://pool.btc.com
MIT License
639 stars 408 forks source link

sharelogger模块无法读取kafka中sharelog的消息 #464

Closed BigPingge closed 4 years ago

BigPingge commented 4 years ago

sharelogger配置如下:

kafka = { brokers = "127.0.0.1:9092"; # "10.0.0.1:9092,10.0.0.2:9092,..." };

sharelog_writers = ( { chain_type = "ETH"; //blockchain short name enabled = true; //enable worker data_dir = "/home/btcpool/data/ethsharelog";

solved share topic

# kafka group id (ShareLog writer use Kafka High Level Consumer)
# use different group id for different servers. once you have set it,
# do not change it unless you well know about Kafka.
kafka_group_id = "sharelog_eth_1";
share_topic = "EthShareLog";

# zlib/gzip compression level: -1 to 9.
# -1: defaule level, 0: non-compression, 1: best speed, 9: best size.
# Changing compression level midway and restarting is OK.
compression_level = -1;

} ); 我本地测试的时候，statshttpd模块能读取到sharelog中的消息，但是sharelogger模块却读不到sharelog。启动后的sharelogger日志如下: Log file created at: 2020/08/30 16:32:42 Running on machine: localhost.localdomain Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0830 16:32:42.218600 12600 ShareLoggerMain.cc:187] BTC sharelogger version 2020.06.09-11-fix-jobmaker-kafka-6-g32ceeed I0830 16:32:42.221071 12600 Kafka.cc:394] consumer librdkafka version: 1.5.0 I0830 16:32:42.221369 12600 ShareLoggerMain.cc:236] wait for worker 140666972141312 I0830 16:32:42.221410 12602 ShareLogger.inl:250] setup sharelog consumer... I0830 16:32:42.225553 12602 Kafka.cc:31] RDKAFKA-4-CONFWARN: rdkafka#consumer-1[thrd:app]: Configuration property offset.store.method is deprecated: Offset commit store method: 'file' - DEPRECATED: local file store (offset.store.path, et.al), 'broker' - broker commit store (requires "group.id" to be configured and Apache Kafka 0.8.2 or later on the broker.). I0830 16:32:42.225590 12602 Kafka.cc:498] add brokers: 127.0.0.1:9092 I0830 16:32:42.226933 12602 ShareLogger.inl:257] waiting sharelog messages... 请教一下造成这样的原因

duguyifang commented 4 years ago

从“Configuration property offset.store.method is deprecated:”看起来是kafka版本问题，高版本kafka摒弃了offset.store字段导致。 请使用Kafka: v0.11.0.2。

BigPingge commented 4 years ago

从“Configuration property offset.store.method is deprecated:”看起来是kafka版本问题，高版本kafka摒弃了offset.store字段导致。 请使用Kafka: v0.11.0.2。

你好，我换了还是这样子，日志显示如下： Log file created at: 2020/08/30 17:57:58 Running on machine: localhost.localdomain Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0830 17:57:58.099756 23846 ShareLoggerMain.cc:187] BTC sharelogger version 2020.06.09-11-fix-jobmaker-kafka-6-g32ceeed I0830 17:57:58.110603 23846 Kafka.cc:394] consumer librdkafka version: 0.11.1 I0830 17:57:58.111837 23846 ShareLoggerMain.cc:236] wait for worker 140487331653376 I0830 17:57:58.113745 23853 ShareLogger.inl:250] setup sharelog consumer... I0830 17:57:58.119779 23853 Kafka.cc:498] add brokers: 127.0.0.1:9092 I0830 17:57:58.120785 23853 ShareLogger.inl:257] waiting sharelog messages...

关于libkafka,0.9.1的版本我也测试过了，也是这样。关于配置，我之前是按文档上来的，但是这儿出了问题，改了还是这种情况

BigPingge commented 4 years ago

从“Configuration property offset.store.method is deprecated:”看起来是kafka版本问题，高版本kafka摒弃了offset.store字段导致。 请使用Kafka: v0.11.0.2。

我其他的模块已经都是正常运行了的，就这儿，不知道怎么的，读不到+_+

BigPingge commented 4 years ago

我今天打印了一下kafka日志,内容如下: Log file created at: 2020/08/31 11:14:35 Running on machine: localhost.localdomain Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0831 11:14:35.336042 8980 ShareLoggerMain.cc:187] BTC sharelogger version 2020.06.09-11-fix-jobmaker-kafka-6-g32ceeed I0831 11:14:35.338322 8980 Kafka.cc:418] consumer librdkafka version: 0.9.1 I0831 11:14:35.339896 8980 ShareLoggerMain.cc:236] wait for worker 140483321906944 I0831 11:14:35.340847 8985 ShareLogger.inl:250] setup sharelog consumer... I0831 11:14:35.341224 8985 Kafka.cc:534] add brokers: 127.0.0.1:9092 I0831 11:14:35.341398 8985 Kafka.cc:31] RDKAFKA-7-BROKER: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Added new broker with NodeId -1 I0831 11:14:35.341986 8988 Kafka.cc:31] RDKAFKA-7-BRKMAIN: rdkafka#consumer-1:0/internal: Enter main broker thread I0831 11:14:35.342144 8988 Kafka.cc:31] RDKAFKA-7-STATE: rdkafka#consumer-1:0/internal: Broker changed state INIT -> UP I0831 11:14:35.342269 8989 Kafka.cc:31] RDKAFKA-7-BRKMAIN: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Enter main broker thread I0831 11:14:35.342383 8989 Kafka.cc:31] RDKAFKA-7-CONNECT: rdkafka#consumer-1127.0.0.1:9092/bootstrap: broker in state INIT connecting I0831 11:14:35.343165 8989 Kafka.cc:31] RDKAFKA-7-CONNECT: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Connecting to ipv4#127.0.0.1:9092 (plaintext) with socket 4 I0831 11:14:35.343595 8989 Kafka.cc:31] RDKAFKA-7-STATE: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Broker changed state INIT -> CONNECT I0831 11:14:35.354244 8989 Kafka.cc:31] RDKAFKA-7-CONNECT: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Connected to ipv4#127.0.0.1:9092 I0831 11:14:35.354295 8989 Kafka.cc:31] RDKAFKA-7-CONNECTED: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Connected I0831 11:14:35.354315 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Using (configuration fallback) 0.9.0 protocol features I0831 11:14:35.354331 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature MsgVer1: Produce (2..2) NOT supported by broker I0831 11:14:35.354336 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature MsgVer1: Fetch (2..2) NOT supported by broker I0831 11:14:35.354339 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Disabling feature MsgVer1 I0831 11:14:35.354343 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature ApiVersion: ApiVersion (0..0) NOT supported by broker I0831 11:14:35.354346 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Disabling feature ApiVersion I0831 11:14:35.354349 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature BrokerGroupCoordinator: GroupCoordinator (0..0) supported by broker I0831 11:14:35.354353 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Enabling feature BrokerGroupCoordinator I0831 11:14:35.354357 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature BrokerBalancedConsumer: GroupCoordinator (0..0) supported by broker I0831 11:14:35.354360 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker I0831 11:14:35.354363 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker I0831 11:14:35.354367 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker I0831 11:14:35.354370 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker I0831 11:14:35.354373 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker I0831 11:14:35.355280 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker I0831 11:14:35.355302 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Enabling feature BrokerBalancedConsumer I0831 11:14:35.355309 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature ThrottleTime: Produce (1..2) supported by broker I0831 11:14:35.355314 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature ThrottleTime: Fetch (1..2) supported by broker I0831 11:14:35.355330 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Enabling feature ThrottleTime I0831 11:14:35.355334 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature Sasl: JoinGroup (0..0) supported by broker I0831 11:14:35.355353 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Enabling feature Sasl I0831 11:14:35.355357 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Feature SaslHandshake: SaslHandshake (0..0) NOT supported by broker I0831 11:14:35.355362 8989 Kafka.cc:31] RDKAFKA-7-APIVERSION: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Disabling feature SaslHandshake I0831 11:14:35.355367 8989 Kafka.cc:31] RDKAFKA-7-FEATURE: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Updated enabled protocol features to BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator I0831 11:14:35.355371 8989 Kafka.cc:31] RDKAFKA-7-STATE: rdkafka#consumer-1127.0.0.1:9092/bootstrap: Broker changed state CONNECT -> UP I0831 11:14:35.376204 8987 Kafka.cc:31] RDKAFKA-7-TOPIC: rdkafka#consumer-1New local topic: EthShareLog I0831 11:14:35.376250 8987 Kafka.cc:31] RDKAFKA-7-DESP: rdkafka#consumer-1Adding desired topic EthShareLog [0] I0831 11:14:35.376260 8987 Kafka.cc:31] RDKAFKA-7-FETCHSTART: rdkafka#consumer-1List with 1 partition(s): I0831 11:14:35.376264 8987 Kafka.cc:31] RDKAFKA-7-FETCHSTART: rdkafka#consumer-1 EthShareLog [0] offset INVALID I0831 11:14:35.376271 8987 Kafka.cc:31] RDKAFKA-7-OFFSETFETCH: rdkafka#consumer-1List with 1 partition(s): I0831 11:14:35.376276 8987 Kafka.cc:31] RDKAFKA-7-OFFSETFETCH: rdkafka#consumer-1 EthShareLog [0] offset INVALID I0831 11:14:35.376346 8989 Kafka.cc:31] RDKAFKA-7-UPDATE: rdkafka#consumer-1127.0.0.1:9092/bootstrap: NodeId changed from -1 to 1 I0831 11:14:35.376358 8989 Kafka.cc:31] RDKAFKA-7-UPDATE: rdkafka#consumer-1127.0.0.1:9092/1: Name changed from 127.0.0.1:9092/bootstrap to 127.0.0.1:9092/1 I0831 11:14:35.376366 8989 Kafka.cc:31] RDKAFKA-7-STATE: rdkafka#consumer-1127.0.0.1:9092/1: Broker changed state UP -> UPDATE I0831 11:14:35.376389 8985 ShareLogger.inl:257] waiting sharelog messages... I0831 11:14:35.386641 8989 Kafka.cc:31] RDKAFKA-7-STATE: rdkafka#consumer-1127.0.0.1:9092/1: Broker changed state UPDATE -> UP I0831 11:14:35.387300 8987 Kafka.cc:31] RDKAFKA-7-STATE: rdkafka#consumer-1Topic EthShareLog changed state unknown -> exists I0831 11:14:35.387321 8987 Kafka.cc:31] RDKAFKA-7-PARTCNT: rdkafka#consumer-1Topic EthShareLog partition count changed from 0 to 1 I0831 11:14:35.387329 8987 Kafka.cc:31] RDKAFKA-7-BRKDELGT: rdkafka#consumer-1Broker 127.0.0.1:9092/1 is now leader for topic EthShareLog [0] with 0 messages (0 bytes) queued I0831 11:14:35.387341 8987 Kafka.cc:31] RDKAFKA-7-BRKMIGR: rdkafka#consumer-1Migrating topic EthShareLog [0] from (none) to 127.0.0.1:9092/1 I0831 11:14:35.487464 8989 Kafka.cc:31] RDKAFKA-7-TOPBRK: rdkafka#consumer-1127.0.0.1:9092/1: Topic EthShareLog [0]: joining broker I0831 11:14:40.341668 8987 Kafka.cc:31] RDKAFKA-7-OFFSET: rdkafka#consumer-1Topic EthShareLog [0]: stored off -1001, committted off -1001 I0831 11:14:40.341884 8987 Kafka.cc:31] RDKAFKA-7-OFFSET: rdkafka#consumer-1Topic EthShareLog [0]: setting offset INVALID for commit I0831 11:14:45.342043 8987 Kafka.cc:31] RDKAFKA-7-OFFSET: rdkafka#consumer-1Topic EthShareLog [0]: stored off -1001, committted off -1001 I0831 11:14:45.342242 8987 Kafka.cc:31] RDKAFKA-7-OFFSET: rdkafka#consumer-1Topic EthShareLog [0]: setting offset INVALID for commit I0831 11:14:50.342598 8987 Kafka.cc:31] RDKAFKA-7-OFFSET: rdkafka#consumer-1Topic EthShareLog [0]: stored off -1001, committted off -1001 I0831 11:14:50.342873 8987 Kafka.cc:31] RDKAFKA-7-OFFSET: rdkafka#consumer-1Topic EthShareLog [0]: setting offset INVALID for commit I0831 11:14:55.343077 8987 Kafka.cc:31] RDKAFKA-7-OFFSET: rdkafka#consumer-1Topic EthShareLog [0]: stored off -1001, committted off -1001 I0831 11:14:55.343137 8987 Kafka.cc:31] RDKAFKA-7-OFFSET: rdkafka#consumer-1Topic EthShareLog [0]: setting offset INVALID for commit 有一个地方很奇怪，EthShareLog [0] offset INVALID，这里无效了，导致读不到。我不知道这个偏移量是怎么设置的，源代码我也没有改变