search

bluesky-social / feed-generator

ATProto Feed Generator Starter Kit
MIT License
919 stars 329 forks source link

handleEvent to async to deal with increased firehose volume #81

Open Bossett opened 7 months ago

Bossett commented 7 months ago

With the recent increase in firehose activity, the firehose subscription struggles to keep up in real time. This gets much worse if you are doing more than a simple string match on posts.

To boost throughput significantly, this PR no longer awaits handleEvent in the subscription. Possible side effect is that if the generator is intended to handle every post and can't do that in real time, this just increases load until failure; but I think risk is small (this is the approach I take in https://github.com/Bossett/bsky-feeds without ill effect).

Exception handling has been moved to handleEvent - leaving it in the subscription with this approach makes logging harder to parse.

aendra-rininsland commented 4 months ago

This PR should really be merged, it has improved throughput on every project I've ported it to (3, at last count).