octue / octue-sdk-python

The python SDK for @Octue services and digital twins.
https://octue.com
Other
10 stars 4 forks source link

Switch to single-topic-per-workspace services #638

Closed cortadocodes closed 7 months ago

cortadocodes commented 8 months ago

Epic

End User Goal

Make asynchronous event handling scalable and more flexible.

Overview

Switch to a single-topic-per-workspace infrastructure where all services in a workspace publish and subscribe to a single topic. Subscriptions are filtered to just the events that service requires. This opens up all kinds of ways of consuming events in a more efficient and scalable manner.

In our current specific use-case, this allows a single cloud function (the async events handler) to consume all events in a workspace and store them in BigQuery for later retrieval. Currently, we're blocked by cloud functions only being allowed one subscription while our service infrastructure necessitates a subscription per service.

Contents