test_partitioned_topic_multi_consumer_ingestion fails occasionally

[imotov]

Describe the bug I ran all our tests many time over night to verify my fix, but as a side effect I collected some of other failures. This test failed in 2.4% of the runs.

running 1 test
test source::pulsar_source::pulsar_broker_tests::test_partitioned_topic_multi_consumer_ingestion ... FAILED

failures:

failures:
    source::pulsar_source::pulsar_broker_tests::test_partitioned_topic_multi_consumer_ingestion

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 149 filtered out; finished in 7.03s

thread 'source::pulsar_source::pulsar_broker_tests::test_partitioned_topic_multi_consumer_ingestion' panicked at 'assertion failed: `(left == right)`
  left: `[b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"0\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-0\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"0\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"1\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-0\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"1\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"2\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-0\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"2\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"3\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-0\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"3\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"4\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-0\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"4\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"5\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-0\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"5\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"6\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-0\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"6\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"7\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-0\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"7\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"8\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-0\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"8\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"9\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-0\"}", b"{\"body\":\"Hello, world! This is some test data.\",\"id\":\"9\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}"]`,
 right: `["{\"body\":\"Hello, world! This is some test data.\",\"id\":\"0\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", "{\"body\":\"Hello, world! This is some test data.\",\"id\":\"1\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", "{\"body\":\"Hello, world! This is some test data.\",\"id\":\"2\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", "{\"body\":\"Hello, world! This is some test data.\",\"id\":\"3\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", "{\"body\":\"Hello, world! This is some test data.\",\"id\":\"4\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", "{\"body\":\"Hello, world! This is some test data.\",\"id\":\"5\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", "{\"body\":\"Hello, world! This is some test data.\",\"id\":\"6\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", "{\"body\":\"Hello, world! This is some test data.\",\"id\":\"7\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", "{\"body\":\"Hello, world! This is some test data.\",\"id\":\"8\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}", "{\"body\":\"Hello, world! This is some test data.\",\"id\":\"9\",\"timestamp\":1674515715,\"topic\":\"test-pulsar-source--partitioned-multi-consumer--topic-veB8p-partition-1\"}"]`', quickwit-indexing/src/source/pulsar_source.rs:1150:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[guilload]

Assigning to myself.