epam / cloud-pipeline

Cloud agnostic genomics analysis, scientific computation and storage platform
https://cloud-pipeline.com
Apache License 2.0
144 stars 58 forks source link

Data Storages: Support filter for listing items #3500

Closed ekazachkova closed 2 months ago

ekazachkova commented 2 months ago

The current PR brings a new method POST /datastorage/<ID>/list/filter?path=<relative path>&showVersion=<true|false>&showArchived<true|false>

{
  "dateAfter": "2023-08-14 23:04:15.000",
  "dateBefore": "2023-08-15 00:07:15.000"
  "nameFilter": "substring",  # find case insensitive substring in file name
  "sizeGreaterThan": 0,
  "sizeLessThan": 0
}

This method returns files located at the path prefix that match the requested filters.

Additionally, since the number of files can be large a new system preference storage.listing.filter.items.limit has been introduced (default: 500). This option restricts the number of items requested from cloud provider.

The non-empty nextPageMarker indicates that there are still files that were not included in the response list (it might be worth increasing the storage.listing.filter.items.limit setting).