search

owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.35k stars 2.06k forks source link

occ files:scan's memory consumption grows with amount of files scanned #19115

Closed danimo closed 8 years ago

danimo commented 8 years ago

Expected behavior

The memory consumption of the scanner is constant.

Actual behavior

The memory consumption of the scanner grows with the amount of files, requiring to increase the memory limit again and again for users with a large amount of data.

Details

ownCloud: 8.1.3 Total amount of file in user's share: 45308 Memory required: > 256 MB, < 512 MB

karlitschek commented 8 years ago

@icewind1991

oparoz commented 8 years ago

It's by design https://github.com/owncloud/core/pull/10922. All the information is collected first, before being sent to the database, in order to speed up the process. So the solution seems to be to give large amounts of memory (in the GB range) to the cli.

Note: Scanned folders are unavailable while the command runs

oparoz commented 8 years ago

I see one possible enhancement: Collect structure data first and launch non-recursive scans from the bottom of the tree. Each folder update will be committed separately.

PVince81 commented 8 years ago

This could be due to stat caches.

When scanning a local storage, PHP will store stuff in its stat cache. Not sure if that one is capped.

Regarding SMB, we added a cap to the stat cache recently: https://github.com/owncloud/core/pull/21648

If this is still happening in 8.2.2 or 9.0 RC1, please reopen.

lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.