Broadcast Stability - Githubissues

inhindsight / hindsight

Apache License 2.0

12 stars 8 forks source link

Broadcast Stability #193

Open ManApart opened 4 years ago

ManApart commented 4 years ago

In sending 10 million small messages, Broadcast died 7 times with OOMKill. Clients (home built, wsta, websocat) were unable to maintain connections even when broadcast was not rolling.

[ ] Broadcast can deliver 10 million messages to at least one client without disconnecting or dying
[ ] Broadcast should never die because of out of memory (proper constraints should be set, everything should stream)
[ ] Broadcast should not disconnect clients

jeffgrunewald commented 4 years ago

@ManApart or @jmorris-pillar can either of you provide a little more context on what conditions these were running on (cloud or local? cluster size?) for reproducibility? Also how the million small messages were generated?

ManApart commented 4 years ago

See @jmorris-pillar for a link to the spreadsheet with test results (not sure if I still have access, I can look too if needed). It gives more detailed numbers. We generated with Insense. For broadcast we let all the messages pile up on the input topic, and then turned it on, so the only limitation was broadcast and kafka.