search

segmentio / topicctl

Tool for declarative management of Kafka topics
MIT License
580 stars 54 forks source link

get offsets count is off by 1 for each partition #144

Closed joshua-courtney-okcupid closed 5 months ago

joshua-courtney-okcupid commented 1 year ago

when I use topicctl get offsets the total count is off by the number of partitions. I suspect that that the number of messages in each partition is being calculated as last_offset - first_offset when it should be last_offset - first_offset + 1

Here's an example output. I know that I just put 100,000 test messages into this topic, but the total only shows 99900

[0]:~/kafka $ topicctl get offsets queue.purge.delete  --cluster-config=cluster-config/dev/cluster.yaml
[2023-06-13 07:59:59]  INFO Partition bounds for topic queue.purge.delete:
------------+--------------+----------------------+-------------+----------------------+----------+----------+-----------
  PARTITION | FIRST OFFSET |      FIRST TIME      | LAST OFFSET |      LAST TIME       | MESSAGES | DURATION | AVG RATE
------------+--------------+----------------------+-------------+----------------------+----------+----------+-----------
  0         | 2955         | 2023-06-13T11:55:20Z | 3939        | 2023-06-13T11:55:21Z | 984      | 373ms    | 2638/sec
  1         | 2988         | 2023-06-13T11:55:20Z | 3983        | 2023-06-13T11:55:21Z | 995      | 373ms    | 2667/sec
  2         | 2997         | 2023-06-13T11:55:20Z | 3995        | 2023-06-13T11:55:21Z | 998      | 373ms    | 2675/sec
  3         | 3075         | 2023-06-13T11:55:20Z | 4099        | 2023-06-13T11:55:21Z | 1024     | 373ms    | 2745/sec
  4         | 2943         | 2023-06-13T11:55:20Z | 3923        | 2023-06-13T11:55:21Z | 980      | 373ms    | 2627/sec
  5         | 3090         | 2023-06-13T11:55:20Z | 4119        | 2023-06-13T11:55:21Z | 1029     | 373ms    | 2758/sec
  6         | 2979         | 2023-06-13T11:55:20Z | 3971        | 2023-06-13T11:55:21Z | 992      | 373ms    | 2659/sec
  7         | 2940         | 2023-06-13T11:55:20Z | 3919        | 2023-06-13T11:55:21Z | 979      | 373ms    | 2624/sec
  8         | 3189         | 2023-06-13T11:55:20Z | 4251        | 2023-06-13T11:55:21Z | 1062     | 373ms    | 2847/sec
  9         | 2742         | 2023-06-13T11:55:20Z | 3655        | 2023-06-13T11:55:21Z | 913      | 373ms    | 2447/sec
  10        | 2985         | 2023-06-13T11:55:20Z | 3979        | 2023-06-13T11:55:21Z | 994      | 373ms    | 2664/sec
  11        | 2958         | 2023-06-13T11:55:20Z | 3943        | 2023-06-13T11:55:21Z | 985      | 373ms    | 2640/sec
  12        | 2892         | 2023-06-13T11:55:20Z | 3855        | 2023-06-13T11:55:21Z | 963      | 373ms    | 2581/sec
  13        | 3081         | 2023-06-13T11:55:20Z | 4107        | 2023-06-13T11:55:21Z | 1026     | 373ms    | 2750/sec
  14        | 3018         | 2023-06-13T11:55:20Z | 4023        | 2023-06-13T11:55:21Z | 1005     | 373ms    | 2694/sec
  15        | 3129         | 2023-06-13T11:55:20Z | 4171        | 2023-06-13T11:55:21Z | 1042     | 373ms    | 2793/sec
  16        | 3048         | 2023-06-13T11:55:20Z | 4063        | 2023-06-13T11:55:21Z | 1015     | 373ms    | 2721/sec
  17        | 3093         | 2023-06-13T11:55:20Z | 4123        | 2023-06-13T11:55:21Z | 1030     | 373ms    | 2761/sec
  18        | 3018         | 2023-06-13T11:55:20Z | 4023        | 2023-06-13T11:55:21Z | 1005     | 373ms    | 2694/sec
  19        | 2964         | 2023-06-13T11:55:20Z | 3951        | 2023-06-13T11:55:21Z | 987      | 373ms    | 2646/sec
  20        | 3036         | 2023-06-13T11:55:20Z | 4047        | 2023-06-13T11:55:21Z | 1011     | 373ms    | 2710/sec
  21        | 3147         | 2023-06-13T11:55:20Z | 4195        | 2023-06-13T11:55:21Z | 1048     | 373ms    | 2809/sec
  22        | 3006         | 2023-06-13T11:55:20Z | 4007        | 2023-06-13T11:55:21Z | 1001     | 373ms    | 2683/sec
  23        | 2916         | 2023-06-13T11:55:20Z | 3887        | 2023-06-13T11:55:21Z | 971      | 373ms    | 2603/sec
  24        | 3171         | 2023-06-13T11:55:20Z | 4227        | 2023-06-13T11:55:21Z | 1056     | 373ms    | 2831/sec
  25        | 2979         | 2023-06-13T11:55:20Z | 3971        | 2023-06-13T11:55:21Z | 992      | 373ms    | 2659/sec
  26        | 3171         | 2023-06-13T11:55:20Z | 4227        | 2023-06-13T11:55:21Z | 1056     | 373ms    | 2831/sec
  27        | 2697         | 2023-06-13T11:55:20Z | 3595        | 2023-06-13T11:55:21Z | 898      | 373ms    | 2407/sec
  28        | 2958         | 2023-06-13T11:55:20Z | 3943        | 2023-06-13T11:55:21Z | 985      | 373ms    | 2640/sec
  29        | 2994         | 2023-06-13T11:55:20Z | 3991        | 2023-06-13T11:55:21Z | 997      | 373ms    | 2672/sec
  30        | 2982         | 2023-06-13T11:55:20Z | 3975        | 2023-06-13T11:55:21Z | 993      | 373ms    | 2662/sec
  31        | 3012         | 2023-06-13T11:55:20Z | 4015        | 2023-06-13T11:55:21Z | 1003     | 373ms    | 2689/sec
  32        | 3003         | 2023-06-13T11:55:20Z | 4003        | 2023-06-13T11:55:21Z | 1000     | 373ms    | 2680/sec
  33        | 3018         | 2023-06-13T11:55:20Z | 4023        | 2023-06-13T11:55:21Z | 1005     | 373ms    | 2694/sec
  34        | 2934         | 2023-06-13T11:55:20Z | 3911        | 2023-06-13T11:55:21Z | 977      | 373ms    | 2619/sec
  35        | 2988         | 2023-06-13T11:55:20Z | 3983        | 2023-06-13T11:55:21Z | 995      | 373ms    | 2667/sec
  36        | 2985         | 2023-06-13T11:55:20Z | 3979        | 2023-06-13T11:55:21Z | 994      | 373ms    | 2664/sec
  37        | 3060         | 2023-06-13T11:55:20Z | 4079        | 2023-06-13T11:55:21Z | 1019     | 373ms    | 2731/sec
  38        | 2955         | 2023-06-13T11:55:20Z | 3939        | 2023-06-13T11:55:21Z | 984      | 373ms    | 2638/sec
  39        | 3081         | 2023-06-13T11:55:20Z | 4107        | 2023-06-13T11:55:21Z | 1026     | 373ms    | 2750/sec
  40        | 2934         | 2023-06-13T11:55:20Z | 3911        | 2023-06-13T11:55:21Z | 977      | 373ms    | 2619/sec
  41        | 2952         | 2023-06-13T11:55:20Z | 3935        | 2023-06-13T11:55:21Z | 983      | 373ms    | 2635/sec
  42        | 2970         | 2023-06-13T11:55:20Z | 3959        | 2023-06-13T11:55:21Z | 989      | 373ms    | 2651/sec
  43        | 2979         | 2023-06-13T11:55:20Z | 3971        | 2023-06-13T11:55:21Z | 992      | 373ms    | 2659/sec
  44        | 2967         | 2023-06-13T11:55:20Z | 3955        | 2023-06-13T11:55:21Z | 988      | 373ms    | 2648/sec
  45        | 2868         | 2023-06-13T11:55:20Z | 3823        | 2023-06-13T11:55:21Z | 955      | 373ms    | 2560/sec
  46        | 2928         | 2023-06-13T11:55:20Z | 3903        | 2023-06-13T11:55:21Z | 975      | 373ms    | 2613/sec
  47        | 3003         | 2023-06-13T11:55:20Z | 4003        | 2023-06-13T11:55:21Z | 1000     | 373ms    | 2680/sec
  48        | 2964         | 2023-06-13T11:55:20Z | 3951        | 2023-06-13T11:55:21Z | 987      | 373ms    | 2646/sec
  49        | 3054         | 2023-06-13T11:55:20Z | 4071        | 2023-06-13T11:55:21Z | 1017     | 373ms    | 2726/sec
  50        | 3051         | 2023-06-13T11:55:20Z | 4067        | 2023-06-13T11:55:21Z | 1016     | 373ms    | 2723/sec
  51        | 3045         | 2023-06-13T11:55:20Z | 4059        | 2023-06-13T11:55:21Z | 1014     | 373ms    | 2718/sec
  52        | 3003         | 2023-06-13T11:55:20Z | 4003        | 2023-06-13T11:55:21Z | 1000     | 373ms    | 2680/sec
  53        | 3099         | 2023-06-13T11:55:20Z | 4131        | 2023-06-13T11:55:21Z | 1032     | 373ms    | 2766/sec
  54        | 3003         | 2023-06-13T11:55:20Z | 4003        | 2023-06-13T11:55:21Z | 1000     | 373ms    | 2680/sec
  55        | 3141         | 2023-06-13T11:55:20Z | 4187        | 2023-06-13T11:55:21Z | 1046     | 373ms    | 2804/sec
  56        | 2973         | 2023-06-13T11:55:20Z | 3963        | 2023-06-13T11:55:21Z | 990      | 373ms    | 2654/sec
  57        | 3075         | 2023-06-13T11:55:20Z | 4099        | 2023-06-13T11:55:21Z | 1024     | 373ms    | 2745/sec
  58        | 2895         | 2023-06-13T11:55:20Z | 3859        | 2023-06-13T11:55:21Z | 964      | 373ms    | 2584/sec
  59        | 2967         | 2023-06-13T11:55:20Z | 3955        | 2023-06-13T11:55:21Z | 988      | 373ms    | 2648/sec
  60        | 3012         | 2023-06-13T11:55:20Z | 4015        | 2023-06-13T11:55:21Z | 1003     | 373ms    | 2689/sec
  61        | 3045         | 2023-06-13T11:55:20Z | 4059        | 2023-06-13T11:55:21Z | 1014     | 373ms    | 2718/sec
  62        | 2916         | 2023-06-13T11:55:20Z | 3887        | 2023-06-13T11:55:21Z | 971      | 373ms    | 2603/sec
  63        | 3039         | 2023-06-13T11:55:20Z | 4051        | 2023-06-13T11:55:21Z | 1012     | 373ms    | 2713/sec
  64        | 3033         | 2023-06-13T11:55:20Z | 4043        | 2023-06-13T11:55:21Z | 1010     | 373ms    | 2707/sec
  65        | 3063         | 2023-06-13T11:55:20Z | 4083        | 2023-06-13T11:55:21Z | 1020     | 373ms    | 2734/sec
  66        | 2997         | 2023-06-13T11:55:20Z | 3995        | 2023-06-13T11:55:21Z | 998      | 373ms    | 2675/sec
  67        | 3129         | 2023-06-13T11:55:20Z | 4171        | 2023-06-13T11:55:21Z | 1042     | 373ms    | 2793/sec
  68        | 2913         | 2023-06-13T11:55:20Z | 3883        | 2023-06-13T11:55:21Z | 970      | 373ms    | 2600/sec
  69        | 2916         | 2023-06-13T11:55:20Z | 3887        | 2023-06-13T11:55:21Z | 971      | 373ms    | 2603/sec
  70        | 2934         | 2023-06-13T11:55:20Z | 3911        | 2023-06-13T11:55:21Z | 977      | 373ms    | 2619/sec
  71        | 2985         | 2023-06-13T11:55:20Z | 3979        | 2023-06-13T11:55:21Z | 994      | 373ms    | 2664/sec
  72        | 2865         | 2023-06-13T11:55:20Z | 3819        | 2023-06-13T11:55:21Z | 954      | 373ms    | 2557/sec
  73        | 3042         | 2023-06-13T11:55:20Z | 4055        | 2023-06-13T11:55:21Z | 1013     | 373ms    | 2715/sec
  74        | 3021         | 2023-06-13T11:55:20Z | 4027        | 2023-06-13T11:55:21Z | 1006     | 373ms    | 2697/sec
  75        | 2988         | 2023-06-13T11:55:20Z | 3983        | 2023-06-13T11:55:21Z | 995      | 373ms    | 2667/sec
  76        | 2841         | 2023-06-13T11:55:20Z | 3787        | 2023-06-13T11:55:21Z | 946      | 373ms    | 2536/sec
  77        | 3075         | 2023-06-13T11:55:20Z | 4099        | 2023-06-13T11:55:21Z | 1024     | 373ms    | 2745/sec
  78        | 2973         | 2023-06-13T11:55:20Z | 3963        | 2023-06-13T11:55:21Z | 990      | 373ms    | 2654/sec
  79        | 3060         | 2023-06-13T11:55:20Z | 4079        | 2023-06-13T11:55:21Z | 1019     | 373ms    | 2731/sec
  80        | 2991         | 2023-06-13T11:55:20Z | 3987        | 2023-06-13T11:55:21Z | 996      | 373ms    | 2670/sec
  81        | 2844         | 2023-06-13T11:55:20Z | 3791        | 2023-06-13T11:55:21Z | 947      | 373ms    | 2538/sec
  82        | 3072         | 2023-06-13T11:55:20Z | 4095        | 2023-06-13T11:55:21Z | 1023     | 373ms    | 2742/sec
  83        | 3069         | 2023-06-13T11:55:20Z | 4091        | 2023-06-13T11:55:21Z | 1022     | 373ms    | 2739/sec
  84        | 3117         | 2023-06-13T11:55:20Z | 4155        | 2023-06-13T11:55:21Z | 1038     | 373ms    | 2782/sec
  85        | 2976         | 2023-06-13T11:55:20Z | 3967        | 2023-06-13T11:55:21Z | 991      | 373ms    | 2656/sec
  86        | 2871         | 2023-06-13T11:55:20Z | 3827        | 2023-06-13T11:55:21Z | 956      | 373ms    | 2563/sec
  87        | 3201         | 2023-06-13T11:55:20Z | 4267        | 2023-06-13T11:55:21Z | 1066     | 373ms    | 2857/sec
  88        | 2904         | 2023-06-13T11:55:20Z | 3871        | 2023-06-13T11:55:21Z | 967      | 373ms    | 2592/sec
  89        | 3069         | 2023-06-13T11:55:20Z | 4091        | 2023-06-13T11:55:21Z | 1022     | 373ms    | 2739/sec
  90        | 3108         | 2023-06-13T11:55:20Z | 4143        | 2023-06-13T11:55:21Z | 1035     | 373ms    | 2774/sec
  91        | 2823         | 2023-06-13T11:55:20Z | 3763        | 2023-06-13T11:55:21Z | 940      | 373ms    | 2520/sec
  92        | 3030         | 2023-06-13T11:55:20Z | 4039        | 2023-06-13T11:55:21Z | 1009     | 373ms    | 2705/sec
  93        | 3015         | 2023-06-13T11:55:20Z | 4019        | 2023-06-13T11:55:21Z | 1004     | 373ms    | 2691/sec
  94        | 2928         | 2023-06-13T11:55:20Z | 3903        | 2023-06-13T11:55:21Z | 975      | 373ms    | 2613/sec
  95        | 3015         | 2023-06-13T11:55:20Z | 4019        | 2023-06-13T11:55:21Z | 1004     | 373ms    | 2691/sec
  96        | 2880         | 2023-06-13T11:55:20Z | 3839        | 2023-06-13T11:55:21Z | 959      | 373ms    | 2571/sec
  97        | 2964         | 2023-06-13T11:55:20Z | 3951        | 2023-06-13T11:55:21Z | 987      | 373ms    | 2646/sec
  98        | 3156         | 2023-06-13T11:55:20Z | 4207        | 2023-06-13T11:55:21Z | 1051     | 373ms    | 2817/sec
  99        | 3072         | 2023-06-13T11:55:20Z | 4095        | 2023-06-13T11:55:21Z | 1023     | 373ms    | 2742/sec
------------+--------------+----------------------+-------------+----------------------+----------+----------+-----------
[2023-06-13 07:59:59]  INFO Total bounds for topic queue.purge.delete across all partitions:
-----------------------+----------------------+----------------+----------+-------------
     EARLIEST TIME     |     LATEST TIME      | TOTAL MESSAGES | DURATION |  AVG RATE
-----------------------+----------------------+----------------+----------+-------------
  2023-06-13T11:55:20Z | 2023-06-13T11:55:21Z | 99900          | 373ms    | 267828/sec
-----------------------+----------------------+----------------+----------+-------------
petedannemann commented 1 year ago

Hi @joshua-courtney-okcupid could you open a PR with this fix and write a unit test to reproduce what you are seeing?

petedannemann commented 5 months ago

Fixed by https://github.com/segmentio/topicctl/pull/171