search

apache / rocketmq-client-cpp

Apache RocketMQ cpp client
https://rocketmq.apache.org/
Apache License 2.0
366 stars 158 forks source link

Consumer stuck and then logs stopped updating #318

Closed memocong closed 3 years ago

memocong commented 4 years ago

Description of backgound: a. The rocketmq server is deployed on aws server (linux) in Ireland

b. Only one process is launched to run celery worker within docker (debian9.0) deployed on Aliyun server (Linux 3.10.0-862.14.4.el7.x86_64 x86_64)

c. Every 80 seconds, a task is sent to the celery worker, then the worker will start rocketmq consumer (rocketmq-client-python 2.0.0) to fetch messages from rocketmq server and stop it after running 60 seconds.

I'm new to rocketmq and the intent to using another celery worker to start/stop rocketmq consumer is to avoid stuck or broken or disconnected rocketmq consumer process after running a long time. The fact is that the rocketmq consumer still gets stuck under this design.

Description of issue: After running for 2~3 days, based on the logs in /root/logs/rocketmq-cpp/, the rocketmq consumer can't be started any more because the logs stopped updating and no messages fetched from rockemq server. However, the celery worker still receives regular task and start/stop consumer without any error message, and the connections of sockets on Aliyun server keep increasing which finally triggers "max connection" alert (more than 4000 connections).

Part of the final log from /root/logs/rocketmq-cpp/ until the rocketmq consumer can't be started any more: [2020-Jun-30 15:40:52.808897](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=10], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.808943](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=11], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.808986](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=12], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809030](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=13], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809075](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=14], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809119](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=15], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809238](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=0], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809318](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=1], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809385](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=2], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809448](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=3], its offset is:19026[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809512](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=4], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809577](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=5], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809642](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=6], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809711](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=7], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809775](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=8], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809848](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809921](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=10], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.809985](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=11], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.810050](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=12], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.810113](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=13], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.810184](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=14], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:52.810243](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=15], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.457541](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-c, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.457722](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-d, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.457855](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=0], its offset is:19043[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.457955](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=1], its offset is:19042[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458036](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=2], its offset is:19038[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458113](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=3], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458199](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=4], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458273](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=5], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458349](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=6], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458417](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=7], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458478](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=8], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458538](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458594](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=10], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458650](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=11], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458706](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=12], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458774](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=13], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458865](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=14], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.458942](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=15], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459018](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=0], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459095](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=1], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459180](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=2], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459249](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=3], its offset is:19026[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459319](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=4], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459389](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=5], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459457](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=6], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459523](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=7], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459587](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=8], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459654](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459723](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=10], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459821](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=11], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459908](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=12], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.459980](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=13], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.460069](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=14], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:40:57.460141](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=15], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.457511](info):getConsumerIdList from broker:172.30.17.26:10911[findConsumerIds:1100] [2020-Jun-30 15:41:02.631924](info):range is:2, index is:0, mqAllSize is:2, averageSize is:2, startIndex is:0[allocate:80] [2020-Jun-30 15:41:02.632007](info):getConsumerIdList from broker:172.30.17.26:10911[findConsumerIds:1100] [2020-Jun-30 15:41:02.806446](info):range is:32, index is:0, mqAllSize is:32, averageSize is:32, startIndex is:0[allocate:80] [2020-Jun-30 15:41:02.806584](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-c, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.806673](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-d, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.806769](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=0], its offset is:19043[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.806845](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=1], its offset is:19042[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.806918](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=2], its offset is:19038[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.806987](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=3], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807057](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=4], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807124](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=5], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807200](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=6], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807275](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=7], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807346](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=8], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807412](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807482](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=10], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807543](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=11], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807612](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=12], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807671](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=13], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807733](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=14], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807802](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=15], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807867](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=0], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807926](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=1], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.807998](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=2], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808062](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=3], its offset is:19026[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808128](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=4], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808209](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=5], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808274](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=6], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808338](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=7], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808406](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=8], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808469](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808531](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=10], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808598](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=11], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808671](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=12], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808743](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=13], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808825](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=14], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:02.808891](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=15], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.457528](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-c, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.457688](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-d, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.457800](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=0], its offset is:19043[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.457881](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=1], its offset is:19042[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.457965](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=2], its offset is:19038[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458039](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=3], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458117](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=4], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458206](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=5], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458276](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=6], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458349](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=7], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458420](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=8], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458493](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458609](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=10], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458690](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=11], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458772](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=12], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458842](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=13], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458910](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=14], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.458979](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=15], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459047](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=0], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459112](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=1], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459189](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=2], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459254](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=3], its offset is:19026[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459328](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=4], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459393](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=5], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459458](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=6], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459526](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=7], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459591](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=8], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459656](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459725](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=10], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459796](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=11], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459866](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=12], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459923](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=13], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.459982](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=14], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:07.460042](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=15], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.457777](info):getConsumerIdList from broker:172.30.19.9:10911[findConsumerIds:1100] [2020-Jun-30 15:41:12.633082](info):range is:2, index is:0, mqAllSize is:2, averageSize is:2, startIndex is:0[allocate:80] [2020-Jun-30 15:41:12.633172](info):getConsumerIdList from broker:172.30.19.9:10911[findConsumerIds:1100] [2020-Jun-30 15:41:12.808703](info):range is:32, index is:0, mqAllSize is:32, averageSize is:32, startIndex is:0[allocate:80] [2020-Jun-30 15:41:12.808849](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-c, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.808939](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-d, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809020](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=0], its offset is:19043[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809095](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=1], its offset is:19042[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809184](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=2], its offset is:19038[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809259](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=3], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809327](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=4], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809394](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=5], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809461](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=6], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809528](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=7], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809596](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=8], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809664](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809743](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=10], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809812](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=11], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809879](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=12], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.809947](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=13], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810006](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=14], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810074](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=15], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810137](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=0], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810202](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=1], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810270](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=2], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810340](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=3], its offset is:19026[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810407](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=4], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810472](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=5], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810540](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=6], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810608](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=7], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810677](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=8], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810751](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810820](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=10], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810882](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=11], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.810947](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=12], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.811005](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=13], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.811066](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=14], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:12.811129](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=15], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:14.129396](info):updateTopicRouteInfo: didn't get the session_credentials from any producers and consumers, please re-intialize it if application needs authentication[getSessionCredentialsFromOneOfProducerOrConsumer:1206] [2020-Jun-30 15:41:17.119388](info):updateTopicRouteInfo: didn't get the session_credentials from any producers and consumers, please re-intialize it if application needs authentication[getSessionCredentialsFromOneOfProducerOrConsumer:1206] [2020-Jun-30 15:41:17.457520](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-c, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.457660](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-d, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.457752](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=0], its offset is:19043[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.457837](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=1], its offset is:19042[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.457911](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=2], its offset is:19038[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.457980](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=3], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458057](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=4], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458128](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=5], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458205](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=6], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458277](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=7], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458344](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=8], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458415](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458492](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=10], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458562](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=11], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458630](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=12], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458688](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=13], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458758](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=14], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458824](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=15], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458882](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=0], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.458943](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=1], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459008](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=2], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459080](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=3], its offset is:19026[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459153](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=4], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459221](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=5], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459287](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=6], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459354](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=7], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459419](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=8], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459482](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459548](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=10], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459615](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=11], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459680](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=12], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459749](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=13], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459808](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=14], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.459865](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=15], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:17.467379](warning):Get topic[TBW102] route failed [TOPIC_NOT_EXIST].[getTopicRouteInfoFromNameServer:323] [2020-Jun-30 15:41:22.457641](info):getConsumerIdList from broker:172.30.17.26:10911[findConsumerIds:1100] [2020-Jun-30 15:41:22.512219](info):DefaultMQPushConsumerImpl shutdown[shutdown:399] [2020-Jun-30 15:41:22.512630](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-c, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.512788](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-d, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.512877](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=0], its offset is:19043[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.512953](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=1], its offset is:19042[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513032](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=2], its offset is:19038[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513100](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=3], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513186](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=4], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513253](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=5], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513319](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=6], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513383](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=7], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513445](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=8], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513511](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513627](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=10], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513707](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=11], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513771](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=12], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513835](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=13], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513898](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=14], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.513961](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=15], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514030](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=0], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514086](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=1], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514166](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=2], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514230](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=3], its offset is:19026[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514298](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=4], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514361](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=5], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514427](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=6], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514489](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=7], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514551](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=8], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514613](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514683](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=10], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514745](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=11], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514804](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=12], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514862](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=13], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514922](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=14], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.514980](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=15], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.515038](info):unregisterClient to broker:172.30.17.26:10911[unregisterClient:283] [2020-Jun-30 15:41:22.632286](info):range is:2, index is:0, mqAllSize is:2, averageSize is:2, startIndex is:0[allocate:80] [2020-Jun-30 15:41:22.632382](info):getConsumerIdList from broker:172.30.17.26:10911[findConsumerIds:1100] [2020-Jun-30 15:41:22.806401](info):unregisterClient to:172.30.17.26:10911 success[unregisterClient:293] [2020-Jun-30 15:41:22.806481](info):unregisterClient to broker:172.30.23.58:10911[unregisterClient:283] [2020-Jun-30 15:41:22.861706](error):msg: no consumer for this group, xxxxxxx,error:1,in file </root/OpenSource/Apache/rocketmq-client-cpp/src/MQClientAPIImpl.cpp> line:768[findConsumerIds:1103] [2020-Jun-30 15:41:22.861768](error):[ERROR] Get empty consumer IDs. Consumer Group: xxxxxxx, Topic: xxxxxxx[doRebalance:94] [2020-Jun-30 15:41:22.861790](error):msg: doRebalance the cidAll is empty,error:-1,in file </root/OpenSource/Apache/rocketmq-client-cpp/src/consumer/Rebalance.cpp> line:96[doRebalance:148] [2020-Jun-30 15:41:22.861845](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-c, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.861936](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=%RETRY%xxxxxxx, brokerName=broker-d, queueId=0], its offset is:0[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862019](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=0], its offset is:19043[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862095](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=1], its offset is:19042[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862176](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=2], its offset is:19038[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862246](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=3], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862314](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=4], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862379](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=5], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862447](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=6], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862514](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=7], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862581](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=8], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862647](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862719](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=10], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862789](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=11], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862853](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=12], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862918](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=13], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.862990](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=14], its offset is:19034[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863055](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-c, queueId=15], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863123](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=0], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863191](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=1], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863258](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=2], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863326](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=3], its offset is:19026[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863390](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=4], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863458](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=5], its offset is:19037[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863521](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=6], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863589](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=7], its offset is:19032[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863656](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=8], its offset is:19030[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863724](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=9], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863783](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=10], its offset is:19036[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863856](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=11], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.863926](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=12], its offset is:19028[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.864006](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=13], its offset is:19033[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.864128](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=14], its offset is:19035[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.864211](info):oneway updateConsumeOffsetToBroker of mq:MessageQueue [topic=xxxxxxx, brokerName=broker-d, queueId=15], its offset is:19039[updateConsumeOffsetToBroker:310] [2020-Jun-30 15:41:22.986211](info):unregisterClient to:172.30.23.58:10911 success[unregisterClient:293] [2020-Jun-30 15:41:22.986284](info):unregisterClient to broker:172.30.19.9:10911[unregisterClient:283] [2020-Jun-30 15:41:23.213073](info):unregisterClient to:172.30.19.9:10911 success[unregisterClient:293] [2020-Jun-30 15:41:23.213142](info):unregisterClient to broker:172.30.28.83:10911[unregisterClient:283] [2020-Jun-30 15:41:23.392851](info):unregisterClient to:172.30.28.83:10911 success[unregisterClient:293] [2020-Jun-30 15:41:23.392949](info):clientFactory:10.99.0.12@16#DEFAULT stop consumer_timerOperation[consumer_timerOperation:907] [2020-Jun-30 15:41:23.393181](info):disconnect:172.30.17.26:10911 start. event:0x558c38817670[disconnect:133] [2020-Jun-30 15:41:23.393223](info):disconnect:172.30.17.26:10911 completely[disconnect:136] [2020-Jun-30 15:41:23.393240](info):disconnect:172.30.19.9:10911 start. event:0x558c388153f0[disconnect:133] [2020-Jun-30 15:41:23.393266](info):disconnect:172.30.19.9:10911 completely[disconnect:136] [2020-Jun-30 15:41:23.393283](info):disconnect:172.30.23.58:10911 start. event:0x558c38814d40[disconnect:133] [2020-Jun-30 15:41:23.393307](info):disconnect:172.30.23.58:10911 completely[disconnect:136] [2020-Jun-30 15:41:23.393327](info):disconnect:172.30.28.83:10911 start. event:0x558c38814db0[disconnect:133] [2020-Jun-30 15:41:23.393350](info):disconnect:172.30.28.83:10911 completely[disconnect:136] [2020-Jun-30 15:41:23.393368](info):disconnect:prod-base-rocketmq-nameserver-01-ira.inner.xxxxxxx.com:9876 start. event:0x558c388135b0[disconnect:133] [2020-Jun-30 15:41:23.393391](info):disconnect:prod-base-rocketmq-nameserver-01-ira.inner.xxxxxxx.com:9876 completely[disconnect:136] [2020-Jun-30 15:41:23.393536](error):TcpRemotingClient::stopAllTcpTransportThread End, m_tcpTable:0[stopAllTcpTransportThread:119] [2020-Jun-30 15:41:23.393557](info):MQClientFactory:10.99.0.12@16#DEFAULT shutdown[shutdown:300] [2020-Jun-30 15:41:23.393570](info):MQClientFactory:10.99.0.12@16#DEFAULT destruct[~MQClientFactory:51]

ifplusor commented 4 years ago

Could you post stacktrace for every thread?

memocong commented 4 years ago

Could you post stacktrace for every thread?

Is there any other place to get the stacktrace of every thread?

In "Description of issue", it is clearly explained that the logs stopped updating and the final log is pasted for reference

...the logs stopped updating ...Part of the final log from /root/logs/rocketmq-cpp/ until the rocketmq consumer can't be started any more:...

ifplusor commented 4 years ago

@memocong You can get stacktrace from gdb. I want to know the status of each thread that created by rocketmq.so, and find out why consumer is stuck. You can also try to use re_dev branch, I think this version is more robust.

ifplusor commented 4 years ago

Frequently start and shutdown consumers is a bad way to use, which will cause rebalance and cause duplicated consume.

memocong commented 4 years ago

Frequently start and shutdown consumers is a bad way to use, which will cause rebalance and cause duplicated consume.

@ifplusor , I agree with you. However, is there a recommended way to monitor whether rocketmq consumers get stuck or broken or disconnected? And, if the consumer gets stuck, what action should be taken?

ifplusor commented 4 years ago

Client can automatically recover when disconnected, but no good way to detect stuck. If you have stuck, I think you have encountered a bug.

memocong commented 4 years ago

Client can automatically recover when disconnected, but no good way to detect stuck. If you have stuck, I think you have encountered a bug.

@ifplusor , I see. I will try to run rocketmq consumer without restarting it and monitoring its status for some while.