grantneale / kafka-lag-based-assignor

Kafka partition assignor that distributes lag evenly across a consumer group
Apache License 2.0
12 stars 5 forks source link

Partitions with lag 0 should be distributed evenly across consumers #1

Open grantneale opened 7 years ago

grantneale commented 7 years ago

Potential solutions: 1) break consumer total lag ties by number of partitions assigned to each consumer 2) ensure each consumer gets at least 1 partition 3) Ensure partition count per consumer is equal, or as close as possible, but minimise total lag within this constraint

Also add a unit test to cover this case

Otherwise we get assignments like this: TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID streaming 3 13955389 13955389 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 6 13955380 13955380 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 7 13955386 13955386 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 8 13683972 13683972 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 9 13955385 13955385 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 10 13811288 13811288 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 11 13955394 13955394 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 12 13955389 13955389 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 13 13710084 13710084 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 14 13955394 13955394 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 15 13955400 13955400 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 16 13955395 13955395 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 17 13955395 13955395 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 18 13955393 13955393 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 19 13803334 13803334 0 consumer-1-a12fdbdc-fff5-4eec-a98d-80ebe7f85cec /103.61.225.28 consumer-1 streaming 0 13686895 13955393 268498 consumer-1-44547b75-bd08-452a-be23-238927608690 /103.61.225.27 consumer-1 streaming 1 13480961 13750104 269143 consumer-1-2a0bce77-28f4-4bb5-a68c-bdac20dd3dc7 /103.61.225.30 consumer-1 streaming 5 13884987 13892084 7097 consumer-1-8238b24c-0f55-40e4-9eaa-f6e69b58a0e5 /103.61.225.35 consumer-1 streaming 4 13951072 13955395 4323 consumer-1-86d5db03-5d62-4f7a-99ce-dd9c82623773 /103.61.225.34 consumer-1 streaming 2 13459110 13732465 273355 consumer-1-26faaaa0-8148-43d6-847c-6fb04ebc9a89 /103.61.225.32 consumer-1