Load shedding for event service - Githubissues

planetary-social / nos-event-service

A service which collects various events to pass them to other Nos services e.g. the notification service.

Mozilla Public License 2.0

1 stars 0 forks source link

Load shedding for event service #89

Closed dcadenas closed 4 months ago

dcadenas commented 5 months ago

A load shedding strategy when we are under backpressure instead of full disconnection as we currently do. Currently when we reconnect we grab 1 hour of data from each relay and that's very hard to change and integrate with the backpressure signal from the consumer. Load shedding is easier to skip that hour surge of events on reconnects and also opens the door to future more dynamic/intelligent adaptations under load.

Screenshot 2024-05-20 at 1 32 08 PM

dcadenas commented 5 months ago

I already merged a PR for this. Now just checking because I think the speed of processing events used to be faster

dcadenas commented 5 months ago

I suspect that the surge that made me do this change was related to a surge in profiles (see https://stats.nostr.band/#relay_users):

dcadenas commented 5 months ago

And this is the cause https://ron.stoner.com/I_Gained_1_Million_Followers_in_24_Hours/

dcadenas commented 5 months ago

This is working but I'm letting it run for a while to see metrics before closing.