search

morganstanley / modern-cpp-kafka

A C++ API for Kafka clients (i.e. KafkaProducer, KafkaConsumer, AdminClient)
Apache License 2.0
345 stars 86 forks source link

Integration tests are failing #44

Closed gobprasad closed 3 years ago

gobprasad commented 3 years ago

Hi, I am seeing below test cases are failing. I believe all these test cases are failing because "Topics" are not created automatically.

I am using latest modern-cpp-kafka with librdkafka ver 1.7.0 and C++14 standard.

[----------] Global test environment tear-down [==========] 41 tests from 7 test cases ran. (843153 ms total) [ PASSED ] 24 tests. [ FAILED ] 17 tests, listed below: [ FAILED ] KafkaAutoCommitConsumer.OffsetCommitAndPosition [ FAILED ] KafkaManualCommitConsumer.NoOffsetCommitCallback [ FAILED ] KafkaManualCommitConsumer.OffsetCommitCallback [ FAILED ] KafkaManualCommitConsumer.OffsetCommitCallbackTriggeredBeforeClose [ FAILED ] KafkaManualCommitConsumer.OffsetCommitCallback_ManuallyPollEvents [ FAILED ] KafkaManualCommitConsumer.OffsetCommitAndPosition [ FAILED ] KafkaManualCommitConsumer.OffsetsForTime [ FAILED ] KafkaManualCommitConsumer.RecoverByTime [ FAILED ] KafkaSyncProducer.SendMessagesWithAcks1 [ FAILED ] KafkaSyncProducer.SendMessagesWithAcksAll [ FAILED ] KafkaSyncProducer.DefaultPartitioner [ FAILED ] KafkaSyncProducer.TryOtherPartitioners [ FAILED ] KafkaAsyncProducer.MessageDeliveryCallback [ FAILED ] KafkaAsyncProducer.DeliveryCallback_ManuallyPollEvents [ FAILED ] KafkaAsyncProducer.NoBlockSendingWhileQueueIsFull_ManuallyPollEvents [ FAILED ] KafkaAsyncProducer.TooLargeMessageForBroker [ FAILED ] KafkaAsyncProducer.CopyRecordValueWithinSend

Error Log Snippet from test failure

[ RUN ] KafkaAutoCommitConsumer.OffsetCommitAndPosition [2021-05-23 15:38:27.602742] Topic[28ff0948-13ab659a] would be used unknown file: Failure C++ exception with description "2021-05-23 15:38:57.606166: Broker: Unknown topic or partition [3] (modern-cpp-kafka//include/kafka/KafkaProducer.h:547)" thrown in the test body. [ FAILED ] KafkaAutoCommitConsumer.OffsetCommitAndPosition (30005 ms)

ALL other test cases are throwing exception at same place as mentioned in above log.

kenneth-jia commented 3 years ago

So, before running the tests, did you setup the Kafka cluster properly? (and then the integration test would get the Kafka broker addresses from env variable KAFKA_BROKER_LIST) FYI, https://github.com/morganstanley/modern-cpp-kafka/blob/main/.github/workflows/kafka_api_ci_tests.yml#L212 https://github.com/morganstanley/modern-cpp-kafka/blob/main/tests/utils/TestUtility.h#L85

gobprasad commented 3 years ago

Yes, KAFKA_BROKER_LIST env is updated prior to test. 24 tests are passing while 17 tests are failing. Please let me know if you need more info.

kenneth-jia commented 3 years ago

I guess your Kafka cluster is refusing the topic auto-creation! See, https://github.com/morganstanley/modern-cpp-kafka/blob/main/doc/KafkaBrokerConfiguration.md#broker-settings

gobprasad commented 3 years ago

Yes, indeed my new Kafka cluster was not having auto-topic-creation enabled. Once I enabled, all tests are passing -

[----------] Global test environment tear-down [==========] 41 tests from 7 test cases ran. (525964 ms total) [ PASSED ] 41 tests.

kenneth-jia commented 3 years ago

Considering some users might be using pre-deployed Kafka clusters (which disabled the auto.create.topics.enable), I just merged a PR to make it not rely on the "topic auto-creation" behavior, https://github.com/morganstanley/modern-cpp-kafka/pull/45 @gobprasad you could pull the latest update and try again. 😉