Distributed: Investigate search throttling

[JurgenR]

There is some kind of throttling going on in the distributed searches which is obvious when looking at the logs. There is often a batch of several distributed search requests in a row within a few milliseconds of each other, then suddenly nothing for a few 100 milliseconds to almost a full second before the next one.

I figured out that the rate is actually about either 20 searches per second or roughly 1000 bytes per second. The rate seems to be the same for at least the distributed branch level that I verified (2, 5, 6), however level 0 and 1 really need to be verified as the rate could flatten out after a certain branch level.

This could have causes:

• library: it's caused by some setting in asyncio and thus I'd prefer this to be fixed. I doubt this is the cause
• client: it's some kind of throttling in other clients. In this case it might need some implementation in the library to mimic this. Based on the fact that the rate does not seem to be related to the distributed branch level this might not be the case either
• server: this seems the most likely one. The server batches the search requests and distributes them evenly across the branch roots. In this case there is nothing to do other than to document it

[JurgenR]

At branch level 4, I saw an average of 15 searches per second, +- 750-760 bps