User storage

[mzur]

We got the idea to implement a new module for self-service user storage. This would allow users to upload files in BIIGLE and use them for new volumes. This is the concept for now:

• A prominent notice/button is added to the create volume form that redirects users to a new page where they can create "storage requests". Maybe implement this as button group: "remote files" opens the URL input field, "uploaded files" opens the volume file browser, "upload new files" redirects to the page for a new storage request.
• The users can upload files as part of the storage request. Once all files were uploaded, the storage request is submitted.
• Each new storage request must be reviewed by instance admins (to prevent the upload of undesired files). They get an email (similar to user registration confirmations) and can review the individual files if they want.
• If a storage request is approved, the user is notified that the files are available for use. They can conveniently choose any approved files in the volume file browser for new volumes. The files of rejected storage requests are deleted (and the user is notified with a "reason").
• Each user has a storage quota (e.g. 50 GB). Files of both pending and approved storage requests are counted. A storage request that exceeds the quota is denied.
• Exceptions can be made for individual users who require an increased quota.
• Users can delete their uploaded files in a dedicated view. They can either delete individual files or whole storage requests (and all associated files). More operations such as move/rename or even a simple download should not be provided. It should be discouraged to use BIIGLE as a primary data storage for the files.
• If a user is deleted, all files associated with their storage requests are deleted as well (if they still exist). I.e. the user directory is deleted in the two storage disks (see below).
• Storage requests have an expiration date (e.g. 12 months). Users get a notification 1 month before expiration, 1 week before expiration and when the request is expired (explaining that they have a grace period of 1 month). Storage requests (and all associated files) are automatically deleted 1 month after expiration.
  • If all files associated with the storage request were already deleted, the storage request is silently deleted 1 month before expiration (when the notification should be sent).
• Users can apply for an extension of the expiration date of a storage request (by another 12 months). Admins get an email to confirm or deny the expiration.

The whole process must be easy for users! Maybe new storage requests can be coupled with the creation of a new volume? If the request is approved, the volume is automatically created.

Technical details:

• Storage requests are implemented as new models that are stored in the DB and associated with the user who created them.
• Two storage disks can be provided for the storage request. One stores the pending requests and one stores the approved files. Only the second disk is available in the volume file browser.
• Files of an approved storage request are moved from one disk to the other.
• Storage for individual users can be implemented with "virtual" storage disks. Files of a user are stored in a directory (e.g. the user ID). The virtual storage disk is configured with the directory prefix, so only files of the user are visible.
• The upload quota is set as a config value. It can be overridden with a value in the user attrs. The value can be updated by global admins via an API request.
• The used storage is also stored in the user attrs. All file operations (upload/delete) modify this value.
• The upload does a MIME type check to only allow compatible image/video files.
• A storage request consists of one or more directories of files. Users can add a (virtual) directory in the UI to create the storage request and then select the files that belong to the directory. Many directories can be added this way. Initially, it must be checked if any of the directory names are already used in the user's storage. Each file is uploaded with the directory name as prefix and stored in that directory in the user's storage disk.
• Add instructions to configure upload_tmp_dir (maybe also open_basedir), post_max_size and upload_max_filesize for large file uploads (e.g. up to 10 GB per file). The temporary files must be stored on a disk with enough space. Configurable maximum file size validation is separate from the PHP ini configuration.

[mzur]

Todo list (biigle/user-storage):

• [x] Implement the StorageRequest model
• [x] API create of storage requests
• [x] API destroy of storage requests (with all files)
• [x] API upload of files for a storage request
• [x] API approve of storage requests
  • by admins
  • send notification
  • set expires_at based on config
• [x] API reject of storage requests (send notification with reason)
• [x] API update of storage requests (i.e. submit for review, set submitted_at)
• [x] API destroy of files of a storage request
• [x] API destroy of directories of a storage request
• [x] API to extend lifetime of storage request (unlimited for now)
• [x] Scheduled job to check for expired storage requests (sends notifications)
  • Offer button to view storage request (there, it can be extended with a click)
• [x] Scheduled job to delete expired storage requests
• [x] Scheduled job to delete stale storage requests (created but not submitted)
• [x] View to create storage requests
• [x] View to show all (pending) storage requests
  • Also shows all uploaded files in a file browser view
  • merge files of all storage requests?
  • requests/files/directories can be deleted
  • files cannot be downloaded here!
  • show used and available quota
  • add link to StorageRequestApproved notification
  • allow extension of requests that are about to expire
• [x] View to review a storage request (for admins)
  • lists all directories/files
  • files can be viewed/downloaded
  • offers approve/reject
  • must handle requests that were deleted in the meantime
• [x] Override FilesystemManager to dynamically resolve user-xxx storage disks
• [x] Extend storage disk authorization gate (biigle/core#438) to allow own user disk
• [x] Integration in create and edit volume views
  • Extend button group of biigle/core#438 with "uploaded files" button (shows file browser)
• [x] Prevent page unload while a new storage request is uploading
• [x] Notification of a submitted storage request (with link to approve or reject view)
• [x] Notification of an approved storage request (with link to view storage request)
• [x] Notification of a storage request about to expire (with link to view storage request)
• [x] Configuration of storage request notification delivery channels (mail or web, like reports/maia etc.)
• [x] Admin area panel showing the total used user storage
• [x] Configurable maintenance mode (like with MAIA) where only admins are allowed to create storage requests
• [x] Console command to migrate existing user storage directories
• [x] Check privacy policy and terms of use
  • Maybe add checkbox to confirm permission to distribute the files as well as terms/privacy
• [x] Write a manual article about storage requests
• [x] Figure out nginx and PHP configuration for file uploads (maximum upload size and tmp directories for both)