search

epam / cloud-pipeline

Cloud agnostic genomics analysis, scientific computation and storage platform
https://cloud-pipeline.com
Apache License 2.0
144 stars 58 forks source link

Mount storages improvements #3426

Closed ekazachkova closed 5 months ago

ekazachkova commented 6 months ago

This PR provides a couple of features related to mount storages process.

  1. CP_CAP_LIMIT_MOUNTS , CP_CAP_SKIP_MOUNTS and CP_CAP_FORCE_MOUNTS parameters shall respect storage paths.

    Background

    Datastorages may be unregistered and registered again with a new ID.

Usage example:

CP_CAP_LIMIT_MOUNTS=1,2,bucket,3,bucket2/path2

Note: This PR provides changes for mount during the launch run case. pipe-fuse process may fail in running launches due to inability to refresh temporary credentials.

  1. Implement an ability to rewrite mount options for separate launch for specific storages.
    • add CP_CAP_MOUNT_OPTIONS_ prefix to run parameters

      Usage example:

      CP_CAP_MOUNT_OPTIONS_2="mount_path=/mount_point:mount_params=-o nolock"
CP_CAP_MOUNT_OPTIONS_3="mount_path=/another_mount_point"
    • this prefix shall be passed to workers
    • list of such prefixes may be updated via CP_CAP_AUTOSCALE_INHERITABLE_PARAMETER_PREFIXESvar