search

epam / cloud-pipeline

Cloud agnostic genomics analysis, scientific computation and storage platform
https://cloud-pipeline.com
Apache License 2.0
145 stars 59 forks source link

Introduce basic support for disk buffering in pipe mount #3411

Closed tcibinan closed 8 months ago

tcibinan commented 10 months ago

Relates #3410.

The pull request brings support for basic disk buffering to pipe mount.

The functionally is optional and is disabled by default. Use the following command to mount a data storage with enabled disk buffering.

pipe storage mount -b storage -o allow_other -m 775 -c read-disk-buffer-path=/tmp/storage-buffer /mount/point

Limitations

  1. Disk buffering is initiated for all files and whole files are written to disk even if only a single byte has been read initially.
  2. Disk buffering does not invalidate on write operations. Each 2 hours all files that are buffered for more than 1 hour are invalidated.