Instead of calling the mqtt handler in the main mqtt subscriber process, we hand off to a sidekiq job that handles the message.
This (based on tests on staging) leads to a 100x speedup in the mqtt subscriber itself, and makes the message processing easily horizontally scalable by adding more sidekiq workers if needed (although currently it doesn't look like we're anywhere close to that, the jobs leave the queue faster than i can see them get added)
Some other improvements:
simplifys a bunch of the retry logic,
add some defensive string sanitising against the unicode errors we've been seeing,
strip out the unneeded websocket push code in the storer classes.
Not merging this as it led to some tricky concurrency problems. Will have to rethink the architecture of the ingester i think to effectively parallelise this
Instead of calling the mqtt handler in the main mqtt subscriber process, we hand off to a sidekiq job that handles the message.
This (based on tests on staging) leads to a 100x speedup in the mqtt subscriber itself, and makes the message processing easily horizontally scalable by adding more sidekiq workers if needed (although currently it doesn't look like we're anywhere close to that, the jobs leave the queue faster than i can see them get added)
Some other improvements: