search

ssbc / go-muxrpc

js/muxrpc in golang (for interfacing ssb/scuttlebot)
MIT License
23 stars 6 forks source link

Reduce goroutine storm #4

Open cryptix opened 5 years ago

cryptix commented 5 years ago

Right now there is no bound on incoming requests. Each new call starts a goroutine.

Especially for legacy ssb replication this means ~9k once the connection is established. Depending on the load of the remote party this happens in a couple of seconds but I‘ve also Seen strained Systems where this built-up takes nearly 15minutes.

cryptix commented 5 years ago

One remedy @keks and I thought about is buffering unhandled requests and draining them on a worker pool of static or scalable size.

The idea is to still drain the incoming conn empty as fast as possible. Otherwise we won’t get answers to our calls.

cryptix commented 5 years ago

The above approach would stall on live querys. For this to be feasable we need some kind of switchboard that can accumulate live querys and serve multiple muxrpc sessions from a worker.

It would drain the query up to the latest and then hand it over to a live drain of the root log.

PragmaticCypher commented 5 years ago

What about making a connection limit configurable, and just dropping/rejecting anything which exceeds it?

cryptix commented 4 years ago

Yup, i thought about a simple token bucket limiter for new calls as well, would be easier to add into the existing code for sure.