search

iggy-rs / iggy

Iggy is the persistent message streaming platform written in Rust, supporting QUIC, TCP and HTTP transport protocols, capable of processing millions of messages per second.
https://iggy.rs
MIT License
1.83k stars 86 forks source link

Write testcase to verify deduplication of messages #92

Open hubcio opened 11 months ago

hubcio commented 11 months ago

Unfortunately we don't have this tested (yet).

0xkelvin commented 11 months ago

@hubcio

i would like to help on this one, i am new with message streaming. silly question : duplication of messages are messages same in content ?

spetz commented 11 months ago

@0xkelvin the message deduplication is an optional feature allowing to ignore the messages with the same ID. Whenever the user sends the messages to the server, there's the optional ID field (u128 e.g. UUID or so) - either the user might provide the unique ID or otherwise the server will generate it automatically. When the deduplication is enabled, the server will check if the received message ID already exists - for example, the message producer might have a bug resulting in sending the same message multiple times or so. If this feature is enabled, then the server doesn't care about duplicated IDs.

0xkelvin commented 10 months ago

i see, let me dive in into the code and try to write the test, it is great if you can help point me to that code file

0xkelvin commented 10 months ago

oh, i think here is it : https://github.com/iggy-rs/iggy/blob/382c972cc9039b157cf87f34370c5eee2feed1ac/server/src/streaming/partitions/messages.rs#L340-L341