Closed kevinwallimann closed 3 years ago
Situation Currently, startingOffsets is set to earliest by KafkaStreamReader if the checkpoint folder doesn't exist, so the query ingests from the beginning at the first microbatch. However, earliest is not set if the already folder exists, but is empty. This is inconsistent behaviour and should be fixed. https://github.com/AbsaOSS/hyperdrive/blob/develop/ingestor-default/src/main/scala/za/co/absa/hyperdrive/ingestor/implementation/reader/kafka/KafkaStreamReader.scala#L98
startingOffsets
earliest
KafkaStreamReader
Desired outcome If the checkpoint folder exists, but is empty, startingOffsets should be set to earliest
Situation Currently,
startingOffsetsis set toearliestbyKafkaStreamReaderif the checkpoint folder doesn't exist, so the query ingests from the beginning at the first microbatch. However,earliestis not set if the already folder exists, but is empty. This is inconsistent behaviour and should be fixed. https://github.com/AbsaOSS/hyperdrive/blob/develop/ingestor-default/src/main/scala/za/co/absa/hyperdrive/ingestor/implementation/reader/kafka/KafkaStreamReader.scala#L98Desired outcome If the checkpoint folder exists, but is empty,
startingOffsetsshould be set toearliest