Scanning slows down after ~10000 files and stuck

[yoav1000]

I used syncthing to backup my entire /storage/emulated/0 (with few ignore patterns). Recently i noticed the folder is constant on scanning mode and never finish, on the web GUI last scan was in 20.1.2021 I suspect its around the time i upgraded to Android 11.

I tried many things including deleting the database, reinstall the app, install @Catfriend1 fork but nothing helped. my android is around 70GB with 55000 files but on rescan after ~16GB and ~10000 files scan become very very slow (one file every few seconds) and never finish.

i suspect its something with Android 11 or bug in the new version of the app as my backup was working flawless for more then a year

I have Updated Samsung S10 with Android 11 tried with V1.14 and @Catfriend1 v1.15-preview.5

[Catfriend1]

Hi,

it is "known" that Android 11 has slower files access than previous versions. Had a link a while ago but currently not at hand. (please Google it) There is no difference in this matter between forked/official, but I'd advice to use stable releases instead of preview builds. Preview is for early testing of new Syncthing code and see if the wrapper needs adjustment.

[AudriusButkevicius]

I don't think this will work in general as an approach. Many things keep databases in /storage/emulated/0, which constantly change, hence you have constant scanning. I don't see anything immediate that needs fixing here, hence if you wish to pursue this further, I suggest you use the forum.

[yoav1000]

The scan of the first ~10000 files (16GB) takes about 30 sec, so I'm not sure file access speed is the issue

[wweich]

Since Android 11 /storage/emulated/0/Android/data/ has several additional protections, which prevents Syncthing from scanning that directory.

[yoav1000]

It might worth mentioning that my ignore pattern exclude the entire /storage/emulated/0/Android folder.

After few tries and errors the problem is with /storage/emulated/0/WhatsApp/Media/WhatsApp Images folder. The folder contains ~30000 .jpg images flat without substantial sub folders. When excluded, scanning the remain 25000 items take 5 minutes but scanning only this folder didn't finish after 6 hours. I also forced stop WhatsApp to make sure its not constant change problem

[Catfriend1]

I recently had a friend trying to sync "WhatsApp/Media" and there was a weird file with .jpg. which caused failure. I've advised him not to sync /WhatsApp

[frikisama]

This issue is still a bug, and maybe should stay open: scanning my Camera directory takes an unreasonable amount of time (measured in hours). What I can't say is if this is an Android bug or a syncthing bug.

About the WhatsApp directory: I too tried to scan it, but after it spending several days (!) trying without giving any error (!), I decided to give up, since the battery life was taking a big hit because of the media scanning.

As it stands right now, syncthing is not really usable for me in LineageOS 18.1 (Android 11), except for some trivial stuff.

[Catfriend1]

@frikisama Please note https://www.reddit.com/r/androiddev/comments/kpn68k/android_11_very_slow_file_access_performance/

[justinormont]

Does Android 12 include a fix for the slow folder enumeration speed?

[Catfriend1]

I don't know anything about 12 yet, maybe you can find some info on google issue tracker where people also reported the 11 slowness.

[yoav1000]

@Catfriend1 I believe i found a clue for the problem. // large directories Walk can be inefficient. https://github.com/syncthing/syncthing/blob/804050259980cfd02fb5d1371728e102b335be4d/lib/fs/walkfs.go#L139

I'm still trying to figure out how the scanning process works but i believe that on big directories Syncthing query the directory over and over, combined with the slowness of Android 11 file access resulting with very very slow scan speed.

Adding a cache or reducing the amount of dirlists for big directories can make a huge improvement for Android 11 users.

To reproduce, create 1 folder with 25,000 files and compare with 25 folders of 1000 files.

[Catfriend1]

@yoav1000 I think your finding might be useful, could you please start a thread on the forum for this, as it's not wrapper related but maybe SyncthingNative can be improved by "core devs".

[imsodin]

What you linked is not related to querying the directory over and over again, quite the contrary: The directory contents are stored in memory, which for large directories can be a lot of memory.
What might be the problem are the case-sensitivity checks, which do list the directory contents more often. And given Android's sdcardfs is case-insensitive, these checks are necessary.

[yoav1000]

@imsodin

Hi, i turned on verbose debugging, DirNames func run before every Open in the big directory inside checkCase. https://github.com/syncthing/syncthing/blob/3967b39a17be2a802703678bf634d869aea662db/lib/fs/casefs.go#L405

If listing the big dir takes more time then 1 sec, won't it invalidate the cache, causing it to list the dir on every file open?

Any reason not to increase the cache timeout and size? https://github.com/syncthing/syncthing/blob/3967b39a17be2a802703678bf634d869aea662db/lib/fs/casefs.go#L22

[imsodin]

Firstly, let me let off some steam (skip at will):
A system that can't list a directory a few times a second is plain pathological, and the mechanism isn't designed for pathological systems (well it is, but for a different pathology).

Secondly, I proposed a change to set the expiry after calling DirNames, thus super-slow devices like this will retain the cache for at least a second afterwards, no matter how long DirNames itself takes. I don't want to add any further complexity with heuristics on the cache duration or similar just for devices which will be slow anyway. If Syncthing can't call DirNames once a second, it probably can't work in any useful way at all: https://github.com/syncthing/syncthing/pull/7794