epam / cloud-pipeline

Cloud agnostic genomics analysis, scientific computation and storage platform
https://cloud-pipeline.com
Apache License 2.0
145 stars 59 forks source link

Support disk buffering for read operations in pipe mount #3410

Open tcibinan opened 10 months ago

tcibinan commented 10 months ago

Background

Currently, pipe mount has memory buffering for read operations. It allows to keep some data in memory and thus reduce a number of requests to an underlying storage. Only a limited amount of data is kept in memory for a single file handle. Once the file handle is closed, then the memory is freed.

Memory buffering improves performance a lot but does not cover all scenarios. A lot of time it is extremely efficient to use disk buffering as well.

Approach

Introduce support for disk buffering in pipe mount.