nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
27.44k stars 4.08k forks source link

Better WebDAV-file-server module as the current one is dirt-slow #47584

Open Ozymandias42 opened 2 months ago

Ozymandias42 commented 2 months ago

Context

I'm using Nextcloud on multiple devices, only some of which I can or am willing to use the client on. Some of them being VMs for which it's easier to just mount via Webdav for the only very occasional need for some file, others are mobile devices on which I use the Cryptomator Client to access individual encrypted folders on my Nextcloud.

As such having accessed Nextcloud from all manner of WebDAV clients I was able to eliminate the clients itself as source for sometimes unreasonably long load times.

Problem

WebDAV access is dirt slow. Especially on reading the directory structure upon entering/loading a folder and then again upon loading or uploading a file.

Test Environment

Production Nextcloud Server with around 5000 files hosted in k3s with traefik and the community docker image From those accessed are around 1600 in a Cryptomator container.
Subsequent test-environments those 1600 are the only ones put there.

WebDAV Clients used over time

Clients included Plasma's Dolphin's KIO framework, the Cryptomator Android app, rclone (thanks to vfs-caching the fastest out of all clients), davfs2, macOS' Finder, Microsoft's Explorer. WebDAV Clients used in the test:

Other WebDAV servers or Nextcloud configurations tested with the same files on the same storage backend as comparison and results:

Protocol Client Result
webdav seafile's seafdav blazingly fast.
webdav Nextcloud test instance with primary storage on s3 via minio just as slow as the main instance
webdav External Storage in Nextcloud on the same minio server accessed via nextcloud's webdav export just as slow as the main instance
s3 direct access to the bucket for the external storage test blazingly fast

Suggestions and Feature request

Use a different module/library/framework for the webdav file server module or change it's implementation. I don't know if this is only the case with the s3 primary storage backend but the database should hold the directory structure and metadata too so why not just query that instead of doing whatever slow process it is currently doing to load files and directory-structure data?

Of the reasons I assume seafile is so fast with webdav, one is it doing just that. Seafile chunks all files into it's own pseudo-s3 object structure (or actual s3 objects in the pro version) and -like nextcloud- stores all metadata and structure data in the database.

With seafile I can scroll through image galleries via webdav and have the images load pretty much as quickly as my connection allows. With Nextcloud each image, even ones just a few kilobytes in size, at the very least a two or more seconds and that would be surprisingly fast. Certainly, however a lot more than would be due to connection limits.

Furthermore even without that chunking, the access times for the external-storage bucket are identical to that of seafile and here the full files and structure is merely transferred into the object tree and not also chunked like seafile does. As S3 and webdav both are pretty much identical on pure read-only access in terms of their REST API I think the comparison is a fair one to make at this point.

SpirioSpirio commented 1 month ago

+1, I have the same need !