search

snarfed / bridgy-fed

🌉 A bridge between decentralized social network protocols
https://fed.brid.gy
Creative Commons Zero v1.0 Universal
748 stars 39 forks source link

Separate service for serving ATProto's `subscribeRepos` #630

Closed snarfed closed 1 year ago

snarfed commented 1 year ago

ATProto BGSes get all of our activity via websocket and the com.atproto.sync.subscribeRepos XRPC subscription. We need to send all new ATProto commits and blocks over that single websocket connection, for all users.

Our current architecture can't do that as is because it autoscales, so we have multiple instances, so not all writes go through a single instance that we can route to BGS subscribers in memory. GAE Standard's manual scaling has specific subdomains for each instance, but not Flex. 😐

Options:

  1. Run a separate Flex service just to serve subscribeRepos.
  2. Fan writes out to all instances to serve over subscribeRepos. How? Eventarc, Cloud Tasks etc are great, but I'm not sure how I'd make sure every instance consumes them and then also that they're marked as completed.
snarfed commented 1 year ago

Progress here! New hub service, I'm enqueuing Cloud Tasks when I write new ATProto commits, and hub is receiving them and enqueuing them for subscribers. Testing is underway.