Asynchronous file operations

[juliusknorr]

Long running file operations like copy or move requests are very prone to timeouts, which means that there might be an inconsistent state if a request gets aborted due to a webserver or php timeout.

Opening this ticket to keep track of further ideas to optimize that in terms that the heavy lifting could be done asynchronously in some background process. Not meant as a concrete implementation suggestion yet.

Some ideas:

• The actual request puts the operation into a queue
• Find a way to handle the pending state in the response
• Allow checking for the progress/state of the operation
  • Maybe have some kind of pending virtual file in the tree (especially for MOVE)

Possible process for a upload:

• Do the chunked upload as before
• The move request just schedules a task in the queue to assemble the file
• As long as the assembly is pending we have a somehow hidden pending file entry
• Clients can check the state on the pending file
• The worker queue will build the final file and move it to the location and remove the pending file

[juliusknorr]

We currentyl already have kind of an API for things like that https://github.com/nextcloud/server/commit/74ae7b8929a7fd3f539fd15efb9533424114a480

For more immediate use in any user triggered action it would require the job execution to be more real-time than a probably running cron job. Some ideas from https://github.com/nextcloud/server/pull/30359#issuecomment-999361487

Might be something related to #29204 to have also an occ command to run individual job classes in addition to the list. This would be similar to how Laravel handles queues laravel.com/docs/8.x/queues some further things to think about

But one thing that also needs to be considered is to have a fallback or failure detection in case the worker is not setup or not processing fast enough.

[juliusknorr]

Some additional context:

WebDAV mentions that there is a 102 Pending http status http://webdav.org/specs/rfc2518.html#STATUS_102 but there doesn't seem to be any reference on how the pending state could be rechecked from the client side.

[tobiasKaminsky]

This is more about REST, but we can do it similar? https://farazdagi.com/posts/2014-10-16-rest-long-running-jobs/

From my understanding it would then be something like

• client triggers long running MOVE (of chunked upload)
• server sends 102 pending
• client can periodically check if this task is done, will either return 102 or success
• client then handles success/error of this long running task

[devnoname120]

@juliushaertl Do you have any updates on this front? 🙏