M5 Filesystem integration

[odeimaiz]

Filesystem integration: In order to facilitate data import (incl. interaction with data that does not reside on the DRC platform) and reduce data transfer bottlenecks, the filesystem of the platform will be redesigned. A framework that supports different levels of caching with hot, warm, and cold storage shall be developed to provide faster access to more frequently required data. Data from S3 and nfs/cifs file systems shall be fully supported. A new file picker service will be established that transparently presents registered storage resources (credentials can be deposited in the user settings), incl. the user’s local machine and network, as if part of a single file-system. Coupled services will access the same mounted resources, eliminating the need of copying the data before being able to access it.

Deliverable: platform service and functionality (Q2)

[mguidon]

Status Report after Rudolph

Hardware #578

• [x] setup of test cluster
• [x] check S3 API compatibility with minio/AWS S3
• [x] integration into master deployment
• [x] benchmarks
• [ ] backup solution
• [ ] block storage provisioning for:
  • [ ] nfs/cifs mounts
  • [ ] file-based databases (dvc, binary repo)

Internals #580

• [x] implement work folder sync with one dynamic sidecar
• [x] add docker plugin to maste
• [ ] test plugin under load over time
• [ ] test cache for multiple users on multiple nodes
• [ ] safe shutdown of sidecar (finishing all file transfer)
• [ ] benchmarks
  • [ ] boot times
  • [ ] large files (>10 GB)
  • [ ] many small files (>10000)
  • [ ] add shared folder for external mounts
  • [ ] ro inputs, rw outputs
  • [ ] mount the whole project ro
  • [ ] all dyn services should have inputs, outputs, scratch and share
  • [ ] investigae 5 GB limit for comp. services (dask-fs)

Mount #579

• [x] Test rclone for sftp, gdrive, dropbox
• [ ] Frontend integration for credentials
• [ ] rclone webfrontend for fuse mount (analogue to dropbox integration in windows)
  • [ ] Translage uuid/nodeid into filenames

[mguidon]

Status Report after Schumann

Hardware #578

• [x] setup of test cluster
• [x] check S3 API compatibility with minio/AWS S3
• [x] integration into master deployment
• [x] benchmarks
• [ ] backup solution
• [ ] block storage provisioning for:
  • [ ] nfs/cifs mounts
  • [ ] file-based databases (dvc, binary repo)

Internals #580

• [x] implement work folder sync with one dynamic sidecar
• [x] add docker plugin to maste
• [x] test plugin under load over time
• [x] test cache for multiple users on multiple nodes
• [x] safe shutdown of sidecar (finishing all file transfer)
• [x] benchmarks
  • [x] boot times
  • [x] large files (>10 GB)
  • [x] many small files (>10000)
  • [ ] add shared folder for external mounts
  • [ ] ro inputs, rw outputs
  • [ ] mount the whole project ro
  • [ ] all dyn services should have inputs, outputs, scratch and share
  • [ ] investigate 5 GB limit for comp. services (dask-fs)

Conclusion: It works smoothly but the docker volume plugin from rclone does not behave nicely. We will have to write our own.

Mount #579

• [x] Test rclone for sftp, gdrive, dropbox
• [ ] Frontend integration for credentials
• [ ] rclone webfrontend for fuse mount (analogue to dropbox integration in windows)
  • [ ] Translage uuid/nodeid into filenames

[mguidon]

Status Report after Ernest Shackleton

Hardware #578

• [ ] backup solution
• [ ] block storage provisioning for:
  • [ ] nfs/cifs mounts
  • [ ] file-based databases (dvc, binary repo)

Internals #580

• [x] Identified a problem in osparc
• [ ] Bug in the rclone docker plugin. Bug report submitted

Mount #579

• [x] Test rclone for sftp, gdrive, dropbox
• [ ] Frontend integration for credentials
• [ ] rclone webfrontend for fuse mount (analogue to dropbox integration in windows)
  • [ ] Translage uuid/nodeid into filenames