Enhance overall VFolder removal experience

[kyujin-cho]

Motivation

File operations take a long time and are very fragile. Backend.AI should assume that vfolder deletion process can fail at any time.

Currently, there are two design issues:

• Purging a user account depends on the deletion of their owned vfolders.
  • If vfolder deletion takes a long time or fails, it may prevent the admins from purging user accounts.
• VFolders in the trash bin should always be restorable.
  • We should hide the vfolders "being deleted" in the filesystem from the users.
  • However, we should show them to the admin so that they can inspect and delete interrupted ones manually.

This is a follow-up to:

• 575 (#713)

• 1892

• 1915

and the completion of:

• 767

Proposed implementation details

Decoupling user account purging and vfolder deletion

• We need to make the user account purging done instantly.
  • [ ] #2405
  • replacing/rewriting #2374
• Either:
  • [ ] Introduce a placeholder user (like @ghost) who owns any vfolder when its owner user is purged.
  • [x] Or, #2404.

Moving to trash bin

• [x] When a user moves a vfolder to trash bin by calling DELETE /vfolders or DELETE /vfolders/{name} API
  • The vfolder state transits to DELETE_PENDING.
  • The filesystem remains unaffected.

Deleting forever

• Upon one of the following events:
  • [x] When the user deletes the vfolder from the trash bin via DELETE /vfolders/delete-from-trash-bin API
  • Including when the admin triggers manual deletion of a vfolder in the trash bin via the control panel
    • [ ] Update the control-panel UI to show the vfolders in the DELETE_{ONGOING|ERROR} state.
    • [ ] Update the control-panel UI to show the orphan vfolders in the DELETE_PENDING state with their owner users/projects gone.
  • [ ] When the user "flushes" the entire trash bin via POST /vfolders/flush,
  • [ ] When the user or an external service triggers immediate deletion of a vfolder via ... API,
  • [ ] When the system triggers periodic cleanup of old vfolders in the trash bin,
  • [ ] Or, when the system triggers periodic cleanup of interrupted vfolder deletions,
• Do:
  • [x] The vfolder state transits to DELETE_ONGOING.
  • [ ] Hide it from the user-observable trash bin as its content is no longer fully recoverable.
  • [ ] The storage-proxy initiates a background filesystem-level deletion task.
  • When this is successfully done, the state now moves to DELETE_COMPLETED.
  • Whne this fails, move the state to DELETE_ERROR (and store the error log somewhere?)
  • Since a vfolder may enter DELETE_{ONGOING|ERROR} states multiple times, we need #2116 as well.

  • 1797 (#1884)

History cleanup

• Never delete the vfolders records during the deletion process. Only delete vfolders in the DELETE_COMPLETE state after the data rentention period.
• User removes database record of VFolder in DELETE_COMPLETED status by calling DELETE /vfolders/purge API
  • [ ] Replace this with backend.ai mgr clear-history.

[fregataa]

We made a glossary related to VFolder deletion.

• move to trash: transit VFolder status to DELETE_PENDING and actual contents remains
• delete forever: delete the actual contents of the VFolder
• restore: transit VFolder status from DELETE_PENDING to READY

Does this issue suggest to replace delete forever with flush or add support for flush?

[achimnol]

Since the filesystem-level deletion process can take a long time, or gets failed and retried afterwards, we should show a small notice to the users like: "The deletion process and restoring the available storage space may take up to 12 hours."