search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.16k stars 3.57k forks source link

msgbacklog is not accurate between shutdowns #15948

Open lujiwen opened 2 years ago

lujiwen commented 2 years ago

Describe the bug the msgBacklog is not accurate after restarts and request msgBacklog for subscription even specify --get-precise-backlog

To Reproduce Steps to reproduce the behavior:

  1. get msgBacklog by 'pulsar-admin topics partitioned-stats --get-precise-backlog --get-subscription-backlog-size " alias ts="pulsar-admin topics stats --get-precise-backlog --get-subscription-backlog-size'
  2. restart broker
  3. request for backlog again, or consume msg in a particular subscirption
  4. we can consume more message than the 'msgBacklog'

Expected behavior the msgBacklog can indicate the precise backlog in a subscription in any case.

Radiancebobo commented 2 years ago

What is your Pulsar version? I didn't reproduce the behavior in v2.9.2

lujiwen commented 2 years ago

@Radiancebobo I am using 2.10.0, I looked into the related source code, but I can not find how pulsar broker persist these metrics

yapxue commented 2 years ago

@lujiwen pulsar count number of entries. Did you enabled batching in producer side?

lujiwen commented 2 years ago

@yapxue yes I did

lujiwen commented 2 years ago

@Radiancebobo @yapxue I see, do you mean the msgBacklog is the number of batches of messages in backlog if I enable batch and send messages in batch from producer to broker? so the actual number of individual messages is batchsize * msgBacklog, right? and the same story for the "msgRateOut".

github-actions[bot] commented 2 years ago

The issue had no activity for 30 days, mark with Stale label.