The Data Broker (DBR) is a distributed, in-memory container of key-value stores enabling applications in a workflow to exchange data through one or more shared namespaces. Thanks to a small set of primitives, applications in a workflow deployed in a (possibly) shared nothing distributed cluster, can easily share and exchange data and messages with a minimum effort. In- spired by the Linda coordination and communication model, the Data Broker provides a unified shared namespace to applications, which is independent from applications’ programming and communication model.
Apache License 2.0
19
stars
23
forks
source link
minimal-copy implementation for read() and get() #93
This is a substantial change to the way responses from Redis are handled and processed.
Instead of a 128MB buffer per socket, it's now only using a 32KB buffer. And in case a response to get or read doesn't fit into 32K, the parser creates a SGE to receive the remaining data directly into the user buffer.
The advantages are:
no more copying of large values
no more limit to 128MB values
much better scalability to larger Redis clusters or many processes per node because of the much smaller memory requirement per socket
Disadvantage: higher complexity of the processing (covering all kinds of combinations of scatter-gather-lists with and without various fractions of partially received data)
I had to reduce the amount of items in SCAN commands (used for dbrDirectory and dbrDelete) because these commands currently break if the list of keys doesn't fit into the receive buffer. This might have a performance impact on the commands because more Redis requests need to be sent. I'll look into ways to mitigate this problem in a future PR.
This is a substantial change to the way responses from Redis are handled and processed. Instead of a 128MB buffer per socket, it's now only using a 32KB buffer. And in case a response to get or read doesn't fit into 32K, the parser creates a SGE to receive the remaining data directly into the user buffer.
The advantages are:
Disadvantage: higher complexity of the processing (covering all kinds of combinations of scatter-gather-lists with and without various fractions of partially received data)
I had to reduce the amount of items in SCAN commands (used for dbrDirectory and dbrDelete) because these commands currently break if the list of keys doesn't fit into the receive buffer. This might have a performance impact on the commands because more Redis requests need to be sent. I'll look into ways to mitigate this problem in a future PR.