WebDAV: Support for fetching folders from very large shares

[DunklerPhoenix]

1. What is not working as documented?

Photoprism can't fetch folder from nextcloud after upgrade to newest Version anymore. It seems it can't get the folderlist.

If I enter the folder manually the sync works

time="2023-05-06T11:00:37Z" level=error msg="service-folder: Propfind \"https://user:***@nextcloud.domain.com/remote.php/dav/files/user\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

2. How can we reproduce it?

Steps to reproduce the behavior:

1. Go to Settings->Services
2. Setup Nextcloud Webdav
3. Enable Sync
4. Photoprism can't fetch folders

3. What behavior do you expect?

Photoprism can fetch folders from Nextcloud

4. What could be the cause of your problem?

idk

5. Can you provide us with example files for testing, error logs, or screenshots?

6. Which software versions do you use?

(a) PhotoPrism Architecture & Build Number: AMD64

(b) Database Type & Version: MariaDB

(c) Operating System Types & Versions: Linux Docker

(d) Browser Types & Versions: Chromium Brave

(e) Ad Blockers, Browser Plugins, and/or Firewall Software? Disabled

Build 230504-cbf48798c

7. On what kind of device is PhotoPrism installed?

(a) Device / Processor Type: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz

(b) Physical Memory & Swap Space in GB 126GB + 256GB swap

(c) Storage Type: SSD

8. Do you use a Reverse Proxy, Firewall, VPN, or CDN?

traefik 2 + bunny

[lastzero]

Guess it's running against a request timeout. Unfortunately, we have already released today, so I can't promise you a quick solution. For the start, it would be good to know how many files and folders you have. Also, you may be able to optimize your Nextcloud instance for better performance with a few config changes.

[DunklerPhoenix]

I don't need a quick fix because if I set the path manually the sync works.

The nextcloud instance is hosted by a big german hoster and is quite quick. Didn't found any settings for webdav optimization there.

The root folder has 9 subfolders, but over all levels the folder should be over 6000 (the sync folder is on level one /photos) The instance had a size of about 1.4 TB

[lastzero]

Ok, so that's pretty big. Setting the path manually seems like a viable solution, since downloading all the folders for autocomplete to work can just take too long with so many folders. 30 seconds or a minute should normally be enough to query the contents of an average-sized WebDAV share, but we can't block the UI forever or make a new request every time the user changes the contents of the input field.

[lastzero]

I think in the old version, fetching folders from very large shares silently failed after a while and required multiple requests, while the current version tries to reduce the number of requests, which then results in no folders showing up in the autocomplete input field instead of at least a few.

[DunklerPhoenix]

Wouldn't it be better to implement that it always request just one level? E.g. You click on the folders field and it shows the root folder entries /

• a
• b
• c
• d

• You choose b and now its shows the sub folders of b. Etc etc /b

• e
• f
• g
• h

You choose h /b/h/

I hope you know what I mean. It's kinda difficult to describe

[lastzero]

https://stackoverflow.com/questions/31284615/meaning-of-depth-header-in-webdav-propfind-method