fablabbcn / smartcitizen-api

The Smart Citizen Engine
https://developer.smartcitizen.me
GNU Affero General Public License v3.0
10 stars 4 forks source link

Refactor MQTT message handling to handle messages asynchronously #325

Closed timcowlishaw closed 3 weeks ago

timcowlishaw commented 1 month ago

Instead of calling the mqtt handler in the main mqtt subscriber process, we hand off to a sidekiq job that handles the message.

This (based on tests on staging) leads to a 100x speedup in the mqtt subscriber itself, and makes the message processing easily horizontally scalable by adding more sidekiq workers if needed (although currently it doesn't look like we're anywhere close to that, the jobs leave the queue faster than i can see them get added)

Some other improvements:

timcowlishaw commented 3 weeks ago

Not merging this as it led to some tricky concurrency problems. Will have to rethink the architecture of the ingester i think to effectively parallelise this