node selects optimal buffer size and serves request within the allocated buffer. Then algorithm is simple:
read buffer from
we can start from the simplest min(MaxObjectSize, PayloadSize | Inf) model and get rid of redundant data copies. This will already reduce resource consumption horizontally. Then we'll develop more adaptive models
Is your feature request related to a problem? Please describe.
ObjectService.Put
devours a lot of memory for data buffers, sometimes excessively. Current scheme:client --->
server OS
|--->gRPC
--->[slicer]
--->ObjectService
---> storageeach stage component of the whole processing buffers the data. This leads to redundancy leading to memory overuse
Describe the solution you'd like
allocate single buffer for the whole operation
where, in general,
storage
are data drives that could be remote (depending on the client-defined storage policy)The buffer size depends on the following factors:
MaxObjectSize
node selects optimal buffer size and serves request within the allocated buffer. Then algorithm is simple:
we can start from the simplest
min(MaxObjectSize, PayloadSize | Inf)
model and get rid of redundant data copies. This will already reduce resource consumption horizontally. Then we'll develop more adaptive modelsDescribe alternatives you've considered
not yet
Additional context
2686
2719