[Bug]: Allowed memory exhausted when listing folder with a large amount of files

[artonge]

Bug description

When listing the files contained in a folder, the following errors will be triggered if the number of file is big (> ~150 000, depends on the memory limit):

Server log:

Allowed memory size of 1073741824 bytes exhausted (tried to allocate 99152104 bytes) at /var/www/nextcloud/apps/dav/lib/Connector/Sabre/PropfindCompressionPlugin.php#68",

Web GUI error:

This directory is unavailable, please check the logs or contact the administrator

The issue mostly exist for the trashbin, but once it exists, the user can't even access its files to clean them up.

Links:

• OC\Files\Cache\Cache::getFolderContentsById

Steps to reproduce

1. Create a lot of files in a user's files directory: for i in {1..150000}; do echo "New File $i" > file$i.txt; done
2. Scan those files: occ files:scan --all
3. Access the user's files with the web UI

Expected behavior

The query should not exhaust the allowed memory by paginating the results.

Nextcloud Logs

{
    "reqId": "ctPxRKNSarVQDP03gQqm",
    "level": 3,
    "time": "2022-06-27T15:27:00+00:00",
    "remoteAddr": "127.0.0.1",
    "user": "admin",
    "app": "PHP",
    "method": "PROPFIND",
    "url": "/remote.php/dav/files/admin/",
    "message": "Allowed memory size of 1073741824 bytes exhausted (tried to allocate 99152104 bytes) at /var/www/nextcloud/apps/dav/lib/Connector/Sabre/PropfindCompressionPlugin.php#68",
    "userAgent": "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36",
    "version": "25.0.0.3",
    "data": {
        "app": "PHP"
    }
}

[iotatau]

I am running into this error as well. I use a Nextcloud snap, version 24.0.3. Is there any way to empty the trash from the command line? Edit: as a workaround I doubled the memory limit for PHP, from 512 to 1024M. https://github.com/nextcloud-snap/nextcloud-snap/issues/959 In my case there were 30 folders and 67139 files in the trash.

[kimmerin]

Hm, I don't think that "paginating results" would work (e.g. the desktop client uses the same method to find files to be synced). Streaming the result should be the best solution and the ability to turn off compression by configuration. The latter would help unless the uncompressed PROPFIND-result is getting bigger than the allowed memory size, though.

[szaimen]

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

[solracsf]

Problem still present.

{
  "reqId": "JmW6z0haJkTbbYp89wWy",
  "level": 3,
  "time": "2023-02-16T12:28:54+01:00",
  "app": "PHP",
  "method": "PROPFIND",
  "url": "/remote.php/dav/trashbin/xxxxx/trash",
  "message": "Allowed memory size of 536870912 bytes exhausted (tried to allocate 158008736 bytes) at /var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php#1649",
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36",
  "version": "24.0.9.2",
  "data": {
    "app": "PHP"
  },
  "id": "63ee13771b264"
}

[skjnldsv]

Fixed with 28

[skjnldsv]

Wrong issue, this is about backend! :robot: