search

d4rken-org / sdmaid-se

SD Maid 2/SE is Android's most thorough cleaning tool.
https://play.google.com/store/apps/details?id=eu.darken.sdmse
GNU General Public License v3.0
2.73k stars 424 forks source link

Storage analysis hangs at scanning WhatsApp #978

Closed Girofox closed 6 months ago

Girofox commented 9 months ago

Describe the bug When starting storage analyser, the scan hangs at WhatsApp. Even waiting 20 minutes doesn't help. My WhatsApp folder is very large with 13 GB but is on internal storage which should be fast. I already did excluded WhatsApp under app exclusions (path exclusion didn't worked because this triggers a scan).

Im SD Maid 1 this isn't a problem, the scan just takes a minute.

To Reproduce Steps to reproduce the behavior:

  1. Go to 'Storage analysis
  2. Click on internal storage / main storage
  3. Wait for scan to start
  4. Scan hangs at WhatsApp

Expected behavior Scan should resume after some time. Maybe there should be an option to exclude individual folders for storage analysis.

Smartphone (please complete the following information):

Screenshots Screenshot_20240128_205435_SD Maid SE

Additional context WhatsApp folder at 13 GB which contains many video files. Can't add the path exclusion because this triggers a scan which causes the hang.

d4rken commented 9 months ago

Can you provide a debug log of this?

I'm preparing some performance improvements here but i'm not sure if that covers your case.

Girofox commented 9 months ago

Can you provide a debug log of this?

I'm preparing some performance improvements here but i'm not sure if that covers your case.

This is the part of the log file where it gets stuck (note the size of WhatsApp Images folder:

2024-01-29T12:36:01.683Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Profile Photos), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-16T22:25:22Z, target=null)
2024-01-29T12:36:01.683Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Images), fileType=DIRECTORY, size=532480, modifiedAt=2024-01-29T08:56:16Z, target=null)
2024-01-29T12:36:05.361Z  V/SDMSE:Gateway:Local: lookupFiles(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media/WhatsApp Images)
Girofox commented 9 months ago

And here is the beginning section of WhatsApp scan. There are some error messages but don't know if they are relevant. Maybe i can provide the same debug log for SD Maid 1 as a comparison.

2024-01-29T12:36:01.158Z  V/SDMSE:SAF:Mapper: Trying to match volume StorageVolumeX(uuid=null, directory=/storage/emulated/0, userlabel=Interner Speicher, volumeX=StorageVolume: Interner Speicher, rootUri=content://com.android.externalstorage.documents/root/primary) against LocalPath(/storage/emulated/0/Android/data/com.whatsapp)
2024-01-29T12:36:01.159Z  V/SDMSE:SAF:Mapper: Trying to match volume StorageVolumeX(uuid=7F0A-229A, directory=/storage/7F0A-229A, userlabel=SD-Karte, volumeX=StorageVolume: SD-Karte (7F0A-229A), rootUri=content://com.android.externalstorage.documents/root/7F0A-229A) against LocalPath(/storage/emulated/0/Android/data/com.whatsapp)
2024-01-29T12:36:01.159Z  V/SDMSE:SAF:Mapper: Target storageVolumes for LocalPath(/storage/emulated/0/Android/data/com.whatsapp) is StorageVolumeX(uuid=null, directory=/storage/emulated/0, userlabel=Interner Speicher, volumeX=StorageVolume: Interner Speicher, rootUri=content://com.android.externalstorage.documents/root/primary)
2024-01-29T12:36:01.159Z  V/SDMSE:SAF:Mapper: toSAFPath() LocalPath(/storage/emulated/0/Android/data/com.whatsapp) -> SAFPath(treeRoot=content://com.android.externalstorage.documents/tree/primary, segments=[Android, data, com.whatsapp])
2024-01-29T12:36:01.160Z  V/SDMSE:Gateway:SAF: No UriPermission match for SAFPath(treeRoot=content://com.android.externalstorage.documents/tree/primary, segments=[Android, data, com.whatsapp])
2024-01-29T12:36:01.160Z  W/SDMSE:Gateway:SAF: lookup(SAFPath(treeRoot=content://com.android.externalstorage.documents/tree/primary, segments=[Android, data, com.whatsapp])) failed.
2024-01-29T12:36:01.163Z  V/SDMSE:Gateway:Local: exists(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp)
2024-01-29T12:36:01.164Z  V/SDMSE:Analyzer:Storage:Scanner:Extensions: Walking content items for LocalPath(/storage/emulated/0/Android/media/com.whatsapp)
2024-01-29T12:36:01.166Z  V/SDMSE:Gateway:Local: lookup(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp)
2024-01-29T12:36:01.169Z  V/SDMSE:Gateway:Local: Looked up: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-16T22:24:51Z, target=null)
2024-01-29T12:36:01.172Z  V/SDMSE:Gateway:Local: lookup(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp)
2024-01-29T12:36:01.176Z  V/SDMSE:Gateway:Local: Looked up: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-16T22:24:51Z, target=null)
2024-01-29T12:36:01.181Z  V/SDMSE:Gateway:Local: lookupFiles(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp)
2024-01-29T12:36:01.188Z  V/SDMSE:Gateway:Local: Looked up 1 items:
2024-01-29T12:36:01.189Z  V/SDMSE:Gateway:Local: #0 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-19T01:51:21Z, target=null)
2024-01-29T12:36:01.189Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-19T01:51:21Z, target=null)
2024-01-29T12:36:01.198Z  V/SDMSE:Gateway:Local: lookupFiles(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp)
2024-01-29T12:36:01.237Z  V/SDMSE:Gateway:Local: Looked up 7 items:
2024-01-29T12:36:01.238Z  V/SDMSE:Gateway:Local: #0 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Backups), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T02:39:28Z, target=null)
2024-01-29T12:36:01.238Z  V/SDMSE:Gateway:Local: #1 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.trash), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T08:56:13Z, target=null)
2024-01-29T12:36:01.238Z  V/SDMSE:Gateway:Local: #2 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.StickerThumbs), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-22T20:11:12Z, target=null)
2024-01-29T12:36:01.238Z  V/SDMSE:Gateway:Local: #3 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Databases), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T02:39:25Z, target=null)
2024-01-29T12:36:01.238Z  V/SDMSE:Gateway:Local: #4 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media), fileType=DIRECTORY, size=4096, modifiedAt=2023-05-25T12:50:29Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Local: #5 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.Thumbs), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-17T07:07:35Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Local: #6 LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.Shared), fileType=DIRECTORY, size=12288, modifiedAt=2024-01-29T12:25:28Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Backups), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T02:39:28Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.trash), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T08:56:13Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.StickerThumbs), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-22T20:11:12Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Databases), fileType=DIRECTORY, size=4096, modifiedAt=2024-01-29T02:39:25Z, target=null)
2024-01-29T12:36:01.239Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/Media), fileType=DIRECTORY, size=4096, modifiedAt=2023-05-25T12:50:29Z, target=null)
2024-01-29T12:36:01.240Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.Thumbs), fileType=DIRECTORY, size=4096, modifiedAt=2021-06-17T07:07:35Z, target=null)
2024-01-29T12:36:01.240Z  V/SDMSE:Gateway:Walker#257401293: Walking: LocalPathLookup(lookedUp=LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.Shared), fileType=DIRECTORY, size=12288, modifiedAt=2024-01-29T12:25:28Z, target=null)
2024-01-29T12:36:01.255Z  V/SDMSE:Gateway:Local: lookupFiles(AUTO->NORMAL): LocalPath(/storage/emulated/0/Android/media/com.whatsapp/WhatsApp/.Shared)
Girofox commented 9 months ago

Update:

i updated to the latest version v0.19.1-beta0 and the scan finishes after 5 minutes. At least it seems to work now.

Still slower compared to SD maid 1 though. The issue seems that my WhatsApp Images folder containd 20k elements inside (not nested though), whose sizes are on average below 500 kB.

Don't know if it is possible to multithread a scan of folder with large number of files. In SD Maid 1 the multithread setting didn't affected media scan.

d4rken commented 9 months ago

Hm 0.19.0 -> 0.19.1 had no changes with regards to SD Maid's IO performance 🤷

Can you record a debuglog for a little bit, with trace mode enabled? I'm wondering where the bottleneck is, the initial File.listFiles() in normal access mode is probably as fast as it gets, if the additional look up of file attributes per File returned by File.listFiles() is the bottleneck, then this could potentially be parallelized, whether that helps depends on the storage hardware 🤔 , might be worth a try tho..

WhatsApp folder at 13 GB which contains many video files. Can't add the path exclusion because this triggers a scan which causes the hang.

Exclusions don't affect the analyzer as it would falsify the size calculations. I have some ideas about changing this top be more "scan subfolders on demand". 🧑‍🏭

Total folder size does not affect performance, but the amount of files does, 20k is quite a lot.

d4rken commented 6 months ago

Closing this as it seems to be fixed. Some performance updates also happened in the meantime.