Large folder shows as empty / loads forever

thenickname commented 4 years ago

Steps to reproduce

Have a large folder on your nextcloud server (in my case 40Gb of data in Photos Folder, a few 1000 files)
Freshly connect Android app to the server
Open the large folder inside the app

Expected behaviour

shows the contained data
at least indicates that viewing such a large folder may take time
behaves as the "Photos" view does: lazy loads data with instant feedback
does not shock the user by saying "hmm your folder is empty", when in fact it contains 40 Gigs of valuable data ;)

Actual behaviour

App says the photos folder is empty (while showing, that it is roughly 40 Gb in size)
as a consequence it is not possible to view the content of the folder
but when viewing photos with the "Photos" view everything shows up just fine

Can you reproduce this problem on https://try.nextcloud.com?

didn't try, but seems reasonable to assume that this is not setup related

Environment data

Android version: 9

Device model: Mi Note 10

Stock or customized system: MIUI11

Nextcloud app version: 3.11.1

Nextcloud server version: 18.0.4snap3

Reverse proxy:

Logs

Web server error log

Insert your webserver log here

Nextcloud log (data/nextcloud.log)

Insert your Nextcloud log here

NOTE: Be super sure to remove sensitive data like passwords, note that everybody can look here! You can use the Issue Template application to prefill some of the required information: https://apps.nextcloud.com/apps/issuetemplate

tobiasKaminsky commented 4 years ago

* does not shock the user by saying "hmm your folder is empty", when in fact it contains 40 Gigs of valuable data ;)

This will be changed with upcoming version.

Lazy loading is unfortunately not possible due to webdav. How many files do yo have in it? I tested it with k5 files and it was shown in <10s.

thenickname commented 4 years ago

Somewhere about 7k. Thing is, it doesn't seem to do a whole lot. Not after seconds or even minutes. At least apparently...

Is it that you'd have to fully sync the folder to see its content in the first place? Doesn't seem to be the case for other folders with less content.

ezaquarii commented 4 years ago

Could be that the request times out? What happens then?

stale[bot] commented 4 years ago

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

thenickname commented 4 years ago

Well to prevent automatic closing of this issue, here's an update: While large folders do not show as empty anymore, they are still not showing the contained files. Loading seems to go on indefinitely.

I've tried splitting a large folder into multiple less large folders but it didn't help much either.

The only way to display large folders seems to be syncing them. But that means, I have to download many GB of data to my phone just to see the content of one folder.

In comparison, when using the browser version, both desktop and mobile will eventually display the folder content. It takes time but at least it works. In the app it doesn't.

stale[bot] commented 4 years ago

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

mikebakke commented 3 years ago

maybe closed but not resolved. I have a large directory - can't immediately say how many files because bizarrely there is no count in the web app. That needs fixing if nothing else.

In my app, I see files with no thumbnails and the refresh attempt starts heavy mysql queries but I get no error messages anywhere.

tobiasKaminsky commented 3 years ago

Folder listing is working with small file count, but at some point it fails? Can you see something in nextcloud.log/apache log on server?

mikebakke commented 3 years ago

Unfortunately not. I've checked error/access logs for nginx, mysql error and the data/nextcloud.log. The only thing I do see is c. 50% load on 2 mysql threads in htop for a sustained period when trying to refresh.

I'm running nginx with fpm and have upped resource limits, mem=900m and timeouts 6000.

To be crystal clear in the folder in question I see good thumbnails for most of the 3000+ files then unresolved thumbnails for the last 50 or so. In a new folder I created, newly added files are added as expected and show good thumbnails.

Yesterday I updated to 20.0.4 and performed all the suggested occ index and big-int mods.

What I will do now is try the nuclear option and fully delete the app from my phone and re-install to see if anything changes. I can't tell at which point the failure occurs for sure but I believe it's at about 3000 files. Current size is c. 3060 and I know that some of those were uploaded from web which is all working as expected.

Will update once I've done that.

Rgds

mikebakke commented 3 years ago

No change unfortunately - all works in smaller directories but the big one will only show blank thumbnails for the 6 files I uploaded after re-installing.

Having done some more testing, I saw the mysql queries running for c. 55 seconds via stopwatch. No matter what limits I changed in mysql or php-fpm ini files that never changed.

I checked all the error logs again - nothing.

What I do see is the following access.log pairs whenever I do the refresh:

192.168.0.134 - mike [18/Dec/2020:10:32:56 +0000] "PROPFIND /remote.php/webdav/farks/ HTTP/1.1" 207 832 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.14.1" 192.168.0.134 - mike [18/Dec/2020:10:33:46 +0000] "PROPFIND /remote.php/webdav/farks/ HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.14.1"

No idea what this means but it correlates closely to my stopwatch 55 seconds at 50 seconds... However, when I do a refresh on my smaller directory I get the same pattern of 2 x PROPFINDs but the second returns 207 rather than 499 so I guess that's pointing towards the issue.

OK, This is client timeout which immediately led me to the app, but I see people talking about proxies and LBs which leads me to think this may be my haproxy setup... I route my https in via that so I wonder if it's that timing out after a minute waiting for NC.

Will see if I can adjust any limits or failing that bypass haproxy and see if that changes anything.

mikebakke commented 3 years ago

Hmm. I eradicated the 499s by changing haproxy timeout values but the webdav PROPFIND still runs for 50s and then stops with no updates passed into the client. The only change is it finishes with a 207, the same as a successful directory, so tidier but not materially better.

Is there any configuration in the app I can look at for webdav or other timeouts? (by app I mean the server side php)

thenickname commented 3 years ago

@mikebakke I'm not sure if you are describing the same issue here. Yours sounds as if it could be server related. My report is specifically for the android app. While large folders are always slow, in the web app their contents show up at some point. The android app fails to show the contents of large folders all together.

This points to an issue with the app, not the server (besides possible optimizations on the server side)

Maybe it makes sense to track your problem as a separate issue?

mikebakke commented 3 years ago

Hi. I don't believe so - the web access works perfectly to view and upload to/from large folders but the app doesn't - it does work fine on smaller folders. Only alternative is that server behaviour is changing when interacting with the app but drilling down is outside my competence so maybe a dev will have a look. If not I will have to amend my storage schemas which is a pain

fracklaus commented 3 years ago

I have a similar issue here. Folders with many photos (several 1000s) in it are very slow to show in the android app, or do not show at all, but only display "loading". Is there any solution/workaround available?

robertugo2 commented 3 years ago

So far only dividing into smaller ones.

czw., 7 sty 2021, 18:28 użytkownik fracklaus notifications@github.com napisał:

I have a similar issue here. Folderd with many photos (several 1000s) in it are very slow to show in the android app, or do not show at all, but only display "loading". Is there any dolution/workaround available?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nextcloud/android/issues/6233#issuecomment-756261448, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQ4YXTNTMX53TFM7A55G7IDSYXVKZANCNFSM4NV6FZ7A .

rarealphacat commented 3 years ago

Same problem here, I have a folder with 100k photos and it doesn't load or at least it took longer than 20mins so I stopped (both PC + android). Then I split them into 5k folders, PC loads flawlessly and android takes 8-10 mins to load.

BLu85 commented 2 years ago

Hi, I experienced the same problem.

Problem: the web page shows a folder with +4500 files, the app shows the same folder with ~3200 pics.

System: Android v7.0 Nextcloud app v3.17.1 Nextcloud server 21.0.4

Description: Had a folder with +4500 photos, 15 GB. After re-installing the app on my phone that folder was shown as empty with a "loading..." info. After more then 10 mins the folder was still empty and there was no activity with the server. The pics are correctly shown in the web page. I decided to move them to another folder and they appeared in it on the app. In the older folder they started reappearing after I had moved more than 2000 pics.

Problem is that the app stopped loading new pics after ~3200 pics. So ~1300 are present in the new folder but not displayed in the app.

Any help? Thanks

MaxRower commented 2 years ago

Same Problem here. Fairphone 3+ Android 10 and Lenovo Yoga Tab 3 Plus LineageOS 18.1 Nextcloud Server 23.0.0 running on Apache/Ubuntu LTS 20.04 with nginx ssl reverse proxy. Nextcloud Android App 3.18.1

nginx logs http status code 499 after 30 or 60 seconds, depending on Android device. It does happen in LAN and via WAN/mobile access. It seems, the nextcloud app simply has a TCP timeout much too small.

192.168.23.188 - xxxxxx [10/Jan/2022:15:05:24 +0100] "PROPFIND /remote.php/dav/files/xxxxxx//photos/Camera/2021/ HTTP/1.1" 207 843 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.18.1" 192.168.23.188 - xxxxxx [10/Jan/2022:15:06:04 +0100] "PROPFIND /remote.php/dav/files/xxxxxx//photos/Camera/2021/ HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Android) Nextcloud-android/3.18.1"

Folder 2021 does contain 4300+ files, mostly JPG.

invario commented 2 years ago

Same here. 8394 files in one of my folders. Works fine on a desktop browser, and also a mobile (Chrome) browser.

The Android app just sits here loading files indefinitely. Switching out of the app seems to stop it from loading in the background. EVENTUALLY, after several minutes, something WILL display, but it still continues to load (the Waiting circle/symbol shows up).

The Android OS even pops up a message saying that the app seems to have stopped responding so I have to keep telling the OS to Wait.

Phone is a Samsung Galaxy GS10+.

This has been happening forever on Nextcloud Android App for me.

Server is Nextcloud v22.2.3 I went through optimizing everything on the server. It's not a server issue as far as I can tell. APCu and Redis are both setup and working.

App is 3.19.0 RC2

Maybe the app is trying to fetch the entire folder listing and displaying it instead of doing pages at a time?

deathblade666 commented 2 years ago

I'm also having this issue only on mobile, but my file count for the folder isn't that high ~700 files, just a larger than my other folders at ~7GB.

fracklaus commented 2 years ago

Any progress on this issue? This one is really annoying, it affects many basic workflows on android. E.g., using Android storage access framework to open or store some files on nextcloud also does not work with large folders.

invario commented 2 years ago

Any progress on this at all? Makes the Android app completely unusable for me. Thanks.

alexmurry84 commented 2 years ago

Hello there. Same problem here. Android App doesn't show photos (displaying "loading") in my InstaUpload/Camera folder synced with my phone album. More than 8k photos and videos. It's all fine only with webgui. Really really annoying. Hope there'll be solutions soon! :)

Using Nextcloud 24 on Truenas 13 and android app 3.20.1 - Sammy S10+

EDIT: after several hours and a lot of curses, photos appeared in the large folder on android app. The problem now is related to stability. In fact, every time I open the folder and try to scroll down, the app often freezes or crashes, making navigation difficult. Other folders work properly.

DistortedBit commented 2 years ago

Lesurglesum commented 1 week ago

Hello, is there an update regarding this problem or plans to fix it?

nextcloud / android