paasovaara
commented
5 years ago Latest thoughts:
- Keep in-memory cache where TTL is the same as the MQTT Broker's TTL
- Let's not use Redis for perf reasons. Even though we're not currently in any hurry, it's in our critical processing path.
- Let's read multiple Pulsar topics in one app in single thread, calc unique hash from the entire message payload using some good and fast hash function (SHA-1?). Then it's fairly generic piece of code
- When we discover a message with hash that has been seen before just throw it away
- In normal case we should have the same amount of forwarded and discarded messages. enable alert if these two differ too much.
- Requires some bootstrapping functionality (Read all old messages for cache TTL) so we don't lose the cache during application restarts. this will always cause a delay to the processing pipeline so it's a bit expensive..
Pulsar log compaction won't work because it keeps the last one and requires manual invocation. Pulsar deduplication won't work since it's only for making sure same consumer message won't be published twice and uses MessageId's for that.
HFP stream contains duplicate messages due to MQTT QoS 1. And if we run multiple instances of pulsar-mqtt-source, we need to deduplicate those streams as well. Keep the first copy of each unique message.
Implement using Pulsar Functions.
Run on Docker host / Docker Swarm using Pulsar Admin API or CLI.
Ask Pulsar devs whether they are interested in: 1) Streaming compaction of topics instead of cron jobs. 2) Compaction by retaining only the first instances of unique messages, not the last.