nextcloud file scanner does not detect endless loops when scanning smb external storage

[toddpfaff]

• create and link to external smb storage on linux samba server
• samba server should have "follow symlinks" enabled (this is samba default)
• create a symlink in the target smb storage location that loops back to itself
• nextcloud file scanner will follow this link indefinitely

I suspect we've been suffering from this problem for a long time now, but I only recently clued in to the problem after our upgrade to NC-12.0.2 when investigating why our oc_filecache table had grown so large (500 million rows!).

Once I noticed the filecache entries corresponding to the smb file path loop, I was able to remove the offending directory structure, and now I'm running 'occ files:scan' processes to clean up the mess. So far oc_filecache size has been reduced from ~500 million to 420 million rows, and I expect this to drop much more.

Please let me know if more information is needed.

[MorrisJobke]

cc @icewind1991 Is this possible at all?

[icewind1991]

Afaik there is no reliable way to distinguish symlinks over smb

[toddpfaff]

symlinks are not the problem, endless looping is. what is needed is something in the file scanner to detect the loop and end scanning, or a nextcloud server-side option to allow a user or administrator to exclude paths from scanning, and perhaps an option to allow each external storage to be included or excluded from scanning.

[reuterbal]

I experience the same with an external SFTP storage. Thus, file scanning never ends and my oc_filecache table grew like hell until I noticed...

[reuterbal]

Is there any workaround for this? Or is it possible to disable file scanning for external storages?

[reuterbal]

For the SFTP-issues, would it help to include the detection of symbolic links for filetype() in the file /apps/files_external/lib/Lib/Storage/SFTP.php? In my eyes, it shouldn't be more than adding

if ((int) $stat['type'] === NET_SFTP_TYPE_SYMLINK) {
    return 'link';
}

As far as I understood, NC doesn't follow symbolic links. Would this efficiently stop the file scanner?

[flapvhd]

Hi all,

As far as I understood, NC doesn't follow symbolic links. Would this efficiently stop the file scanner?

I see the same issue :-( Does you solution resolve the problem?

Thanx for your help.

[reuterbal]

@falpvhd No it did not.

Any news on this? I consider this a bug, as it effectively prohibits using my external storage.

[sergiomb2]

As references of https://help.nextcloud.com/t/fastest-way-to-remove-large-number-of-filecache-entries/20747/11 https://github.com/nextcloud/server/issues/7312 https://github.com/nextcloud/server/issues/20349 https://github.com/nextcloud/server/issues/16834

For some reason , someone add / of linux filesystem to external storage and nc start to scanning /sys and /proc , which is not good as they aren't regular files and I think that enter into an infinite loop ...

[PotatoYummy]

Look, i install the nextcloud on my Qnap NAS, and here is my filecache table What the hell? cleanup not working. The filecache records all the NAS system files which is useless for me. So what should I do now? Maybe uninstallation is a good solution -_-|||