Deleting directory containing a large number of files fails to show progress at the server, and eventually times out

[davidjericho]

Expected behaviour

Delete a directory from within the sync folders on a client, and see progress of files being deleted. No HTTP timeouts should occur. Files should be deleted from the server.

Actual behaviour

A user deletes a directory from within a sync folder. The progress bar does not move while the directory contents are being deleted, and will sometimes timeout triggering another client side scan and the delete to resume or restart depending on the number of files within the directory.

Additional to this, as there's no IO from the server to the client or the other way for an extended period, the HTTP session times out and the client will sometimes get a disconnect. The client window displays as below for 5 minutes.

After the default HTTP timeout in the Apache web server and Haproxy front end (300 seconds), the client disconnects and displays as below.

The problem is more obvious when the the directory contains many hundreds or thousands of files.

If a user enters the directory on the client machine running the sync client, deleting all the files within (rather than deleting the directory from the parent), the delete operation works without error, and displays progress.

In this test case, the files have never deleted from the server in storage, although they are appearing in the trashbin folder. I also have many versions of the deleted files in the trashbin.

There may be server changes needed too, as I believe the root cause is the local storage interface, the rmdir() function at https://github.com/owncloud/core/blob/stable7/lib/private/files/storage/local.php#L41 . When no depth recursion is required, deletes work and progress is reported. When recursion is done server side, it does not work. This same delete problem certainly manifests when deleting from the web interface as the spinning wheel of nothing.

I have to delete files using a webdav client, or by manually deleting them from the server. I can also trigger a delete using the web interface, close the browser window and come back some time later to find the directory removed.

Steps to reproduce

1. Create a directory within a sync folder, and add many hundreds of files to it.
2. Delete the folder on the client machine.
3. Watch the sync client initiate a delete, and see the progress bar not move while deleting the contents.
4. Client will eventually time out if KeepAlive timeouts are reached.

   Server configuration

Operating system: Red Hat Enterprise Linux Server release 6.6 (Santiago)

Web server: RHEL Apache httpd-2.2.15-39.el6.x86_64

Database: MariaDB 10 with GAlera

PHP version: Remi PHP, php56-2.1-1.el6.remi.x86_64

ownCloud version: ownCloud 7.0.6

Storage backend: Local disk

Client configuration

Client version: 1.8.3

Operating system: Windows 8.1

OS language: English (AU)

Installation path of client: C:\Program Files (x86)\ownCloud

Logs

1. Output of owncloud --logwindow or owncloud --logfile log.txt: https://gist.github.com/davidjericho/a29caeca3f92e46ece65
2. Web server error log: No errors web server side, other than connection timeout reported in the layer7 proxy.
3. ownCloud log (data/owncloud.log): https://gist.github.com/davidjericho/fe8366014f15973256bc

[davidjericho]

I just now tested this with 8.1, and up to 10,000 files in 100 nested directories. The fault did not occur with 8.1, being orders of magnitude faster.

[scolebrook]

@davidjericho We have had the same problem with multiple versions of 7. Thanks for posting your experience with 8.1. Very encouraging news.

[guruz]

@davidjericho Thanks for the report. Good to hear that 8.1 fixes this..

Indeed at some point we should switch to more asynchronous operations so a breaking connection during a long running operation doesn't leave the user confused about if it suceeded or not. @DeepDiver1975 @PVince81 @dragotin @ogoffart