tests are flaky, at least on MacOS

[tscholak]

this is me trying to build the following derivation with ghc 8.6.5:

{ mkDerivation, base, bifunctors, bytestring, c2hs, containers
, either, hspec, monad-loops, rdkafka, stdenv, text, transformers, unix
}:
mkDerivation {
  pname = "hw-kafka-client";
  version = "2.6.0";
  sha256 = "1318gyl3jn3q2namzpzf0254hqpib2nn1kipf6gnfp4dvwv0wbgn";
  isLibrary = true;
  isExecutable = true;
  libraryHaskellDepends = [
    base bifunctors bytestring containers text transformers unix
  ];
  librarySystemDepends = [ rdkafka ];
  libraryToolDepends = [ c2hs ];
  testHaskellDepends = [
    base bifunctors bytestring containers either hspec monad-loops text
    transformers
  ];
  description = "Kafka bindings for Haskell";
  license = stdenv.lib.licenses.mit;
}

test results:

Running 2 test suites...
Test suite integration-tests: RUNNING...

Kafka.Integration
  Per-message commit
    Run producer
      1. sends 2 messages to test topic
    Consumer with per-message commit
      2. should receive 2 messages
    Run producer again
      3. sends 2 messages to test topic
    Consumer after per-message commit
      4. should receive 2 messages again
  Store offsets
    Run producer
      1. sends 2 messages to test topic
    Consumer with no auto store
      2. should receive 2 messages without storing
    Run producer again
      3. sends 2 messages to test topic
    Consumer after commit without store
      4. should receive 4 messages and store them
    Run producer again
      5. sends 2 messages to test topic
    Consumer after commit with store
      6. should receive 2 messages
    Part 3 - Consume after committing stored offsets
      5. sends 2 messages to test topic
      6. should receive 2 messages
  Kafka.IntegrationSpec
    Run producer
      sends messages to test topic
    Run consumer
      should get committed
      should get position
      should receive messages
      should get watermark offsets
      should return subscription
      should return assignment
      should return all topics metadata FAILED [1]
      should return topic metadata
      should describe all consumer groups FAILED [2]
      should describe a given consumer group
      should describe non-existent consumer group
      should read topic offsets for time
      should seek and return no error
      should seek to the beginning
      should seek to the end
      should respect out-of-bound offsets (invalid offset)
      should respect out-of-bound offsets (huge offset)
  Kafka.Consumer.BatchSpec
    Batch consumer
      should consume first batch
      should consume second batch with not enough messages
      should consume empty batch when there are no messages

Failures:

  tests-it/Kafka/IntegrationSpec.hs:157:17:
  1) Kafka.Integration.Kafka.IntegrationSpec, Run consumer, should return all topics metadata
       expected: Right 1
        but got: Right 2

  To rerun use: --match "/Kafka.Integration/Kafka.IntegrationSpec/Run consumer/should return all topics metadata/"

  tests-it/Kafka/IntegrationSpec.hs:167:17:
  2) Kafka.Integration.Kafka.IntegrationSpec, Run consumer, should describe all consumer groups
       expected: Right [ConsumerGroupId {unConsumerGroupId = "it_spec_03"}]
        but got: Right [ConsumerGroupId {unConsumerGroupId = "batch-consumer"},ConsumerGroupId {unConsumerGroupId = "it_spec_03"}]

  To rerun use: --match "/Kafka.Integration/Kafka.IntegrationSpec/Run consumer/should describe all consumer groups/"

Randomized with seed 1585632419

Finished in 63.9449 seconds
33 examples, 2 failures

[tscholak]

hm, starting with a fresh Kafka container made the tests pass. could have been luck, too, though

[AlexeyRaga]

Yeah, this is a bit unsatisfactory...

The thing is that the tests are meant to be run in a fresh kafka container. What happens is that a test creates consumer groups as it goes. But when you run it again, the consumer groups are already there, and the offsets are already committed, etc. And consumer groups cannot be deleted in Kafka...

It is inconvenient a bit, and perhaps something can be done about it. For example, generate a new consumer group id every time and make assertions only on that one disregarding all the others...

But it works in CI, because it gets a new Kafka every time ;)

So I think tests are not flaky, they are "just" not idempotent...

[tscholak]

thanks for the clarification!