infobip / popout

Java file-based extremely fast and reliable FIFO queue.
Apache License 2.0
58 stars 13 forks source link

BatchedFileQueue does not persist all data on close #8

Open ozivotsky opened 1 year ago

ozivotsky commented 1 year ago

BatchedFileQueue implementation does not persist head (reading cache) on flush and does not persist it even on close. The head of the queue is kept only in a memory and data are lost when the program exit.

JUnit test simulating the bug:

class BatchedFileQueueTests {

  @BeforeEach
  void beforeEach () {
    clearTestFiles();
  }

  @AfterEach
  void afterEach () {
    clearTestFiles();
  }

  @Test
  void close () {
    val builder = FileQueue.<Character>batched()
        .name("batched-queue-close")
        .restoreFromDisk(true)
        .folder(FOLDER)
        .serializer(Serializer.CHARACTER)
        .deserializer(Deserializer.CHARACTER)
        .wal(WalFilesConfig.builder()
            .maxCount(Integer.MAX_VALUE)
            .build())
        .batchSize(1000);

    val chars = RandomString.make(2000).toCharArray();
    try (val queue = builder.build()) {
      for (val character : chars) {
        queue.add(character);
      }
      val ch = queue.poll();
      queue.add(ch);

      assertThat(queue.size()).isEqualTo(chars.length);
    }

    try (val queue = builder.build()) {
      // AssertionFailedError:
      // Expecting: <1001> to be equal to: <2000> but was not.
      //  -  999 characters were lost !
      assertThat(queue.size()).isEqualTo(chars.length);
    }
  }
}

Suggested solution: flush also BatchedFileQueue.head in close() method the same way as the tail is flushed.