Describe the bug
When using KafkaBackend as the topology backend, in case there are more than 50 messages in the backend topic, the application ignores the state represented in the up-to-date messages in the topic.
To Reproduce
Steps to reproduce the behavior:
Configure a backend topic for JulieOps with one partition only, and retention time -1 (infinite) or compacted topic
Run multiple topologies with different topology keys (at least 100 times - to have 100 messages in the state topic)
Try to run one of the topologies that were recently updated (e.g., the one that was updated on the 100th message)
The application reads only the first 50/75 messages from the state/backend topic and ignores the latest state as it is described in the latest messages with a higher offset. The number of consumed messages can be between 45-75 messages per bulk.
Expected behavior
The application should read all messages before deciding on the current state to act according to the latest state.
Runtime (please complete the following information):
OS: Windows, Alpine, CentOS
JVM version: openjdk 11, java 17
Version 4.4.1
Additional context
In the KafkaBackendConsumer, once the application receives the first bulk of messages, it builds the plan according to it, and ignores the second read. The application should verify first that there are no more messages to read before deciding on the current state.
Describe the bug When using KafkaBackend as the topology backend, in case there are more than 50 messages in the backend topic, the application ignores the state represented in the up-to-date messages in the topic.
To Reproduce Steps to reproduce the behavior:
Expected behavior The application should read all messages before deciding on the current state to act according to the latest state.
Runtime (please complete the following information):
Additional context In the KafkaBackendConsumer, once the application receives the first bulk of messages, it builds the plan according to it, and ignores the second read. The application should verify first that there are no more messages to read before deciding on the current state.