ddubbert
commented
4 years ago There are different possibilities to make sure, that only the last events are considered.
First of all, Log Compaction might be used as described in this stackoverflow comment. In this case, messages of a producer all need to be marked with the same key, so that only the last copy of each would be remained.
A second approach would be to reduce the log.retention.ms or log.retention.minutes properties to a very low value, so that messages get deleted very fast. This might in turn lead to performance losses because of all the additional I/O tasks.
A last approach, which will firstly be used in this project, is to consume all incoming messages and to store them in a local state (overwrite it every time). In this case, the processing of this state should be decoupled from the message reading, so that it can be done in self defined intervals, always with the newest state.
If an event has been published it is important, as long as there is no newer event in that topic. Older events may represent old positions of players or old sizes of them, which are outdated and should not be used anymore. The question is: Is there a possibility to always request the newest event of a topic as a consumer and ignore every event that happened before.