App hanging for no reason

changemenemo commented 3 years ago

Expected behaviour

the app shouldn't be hanging because of an auto-upload or anything like that. It should be compartimentalized and the process should be running fine even if one of the server isn't responding. I don't know what causes this. The only activity running is an auto upload of all of my galleries. The instance is a 4 cores of xeon gold in a cloudamo data center with 10G of ram on ssd storage. So I highly doubt that it's the server fault since the iOS autoupload are working fine. There are still 4G left in internal storage. So I don't know if it's a copy process from one place to another in the internal storage. If it's the access to the file because of a lock of the app that fills this gallery or something else. But if it's a ressource requirement for Android device that needs to be of a certain IPC by core or anything alike, it would be good to have any information about that.

Actual behaviour

it s in the screens hot. And that happens every minute and since I've beginning to use the android device this current year.

Can you reproduce this problem on https://try.nextcloud.com?

It happened on different instances so it's not related to the server most likely.

Environment data

Android version: 10 but happened on version 9 too

Device model: fairphone 3

Stock or customized system: fairphone version

Nextcloud app version: 3.14 but happened on earlier version too

Nextcloud server version: 20.0.2 happened on earlier version too

Reverse proxy: none

Logs

Web server error log

Insert your webserver log here

No errors

Nextcloud log (data/nextcloud.log)

Insert your Nextcloud log here

Don't have several hours to scrub any vital information just yet

NOTE: Be super sure to remove sensitive data like passwords, note that everybody can look here! You can use the Issue Template application to prefill some of the required information: https://apps.nextcloud.com/apps/issuetemplate

tobiasKaminsky commented 3 years ago

When does it hang? Only when changing something in auto upload? Or while browsing files? Or even if the app is in background?

changemenemo commented 3 years ago

Even when the app is in background. So it can happen when I m watching something on Netflix or when I m having a conversation. And since it has only one task which is the auto upload it is related to that. Since I don't know what's the exact process of the auto-upload (if it needs to get a copy of the file and put it in the nextcloud folder then upload it out etc) I wouldn't be able to be more precise than that. It is happening on the foreground also when I want to just see the list of the auto upload, when I want to scroll the list, less when I try other things like getting the document list or others. I will try of course to get the log but no promises.

changemenemo commented 3 years ago

Do you need a video to confirm the amount of time thay it occurs?

changemenemo commented 3 years ago

@tobiasKaminsky as soon as you have some time on your end, could you guide me about the logging procedure? I really want to understand why the app is so slow and determine if it s the device itself or not.

tobiasKaminsky commented 3 years ago

Sorry, I have no time for explaining/helping in this area :S Maybe someone else can help you.

changemenemo commented 3 years ago

For what I have asked on the fairphone community, They all have the same problem when there are a big number of files and that they mix internal and external folder.

Your description in autoupload of each folder is not enough. I have several Download folders. Some from emulated/0 and some from 3674-2923. for what they can tell me they have problems https://forum.fairphone.com/t/is-anyone-using-nextcloud-on-their-fairphone/67939?u=boistordu

and visibly there is a problem of processing the external directories vs the internals one. And I'm still waiting for help for the command with adb under windows to get accurate logs

AndyScherzinger commented 3 years ago

Sorry that I haven't been able to help you with adb/logs on Windows.

I read through the thread you posted which mixes some different issues.

internal/external storage has issues since Android depending on version and OEM might have issues with media indexing / access rights
activating auto upload for a complete storage point will crash the app (when initially activating this for a "folder" we scan the folder for existing files and trigger their upload, for massive folders this usually leads to "application not responding" scenarios

So I fear for both scenarios you won't find much additional information in the logs since the app is either is overload-moad or jsut doesn't detect anything (so also won't write any logs for that)

changemenemo commented 3 years ago

aaaaaah thanks @AndyScherzinger for those clarification. The first point I totally understand but then can I suggest something? I don't know if it's feasable, but I know that even on a non-routed device, my astro file manager is capable of discerning what's on the SD CARD and what's internal memory. If they can, I 'm sure we can do too no? So if we can do that, then if it would be maybe preferable, if we know that we got problems about that, to put those folders in another section, like in a paragraph at the end of the same page, or on a different tab? So that we can put a disclaimer and warn people about the potential problems. Because right now for what I can see, every folders are mixed together/ And also some folders I'm under the impression that this is the same folder but accessing it through the internal memory for example (so through emulated).

For the second point, is it because of the limitations of android itself? Because marino the dev for ios did overcome the problem apparently. And I guess (but I didn't verify the code), that it implies copying in the nextcloud folder of the internal memory a copy or at least the index. And how massive are we talking about? But the application not responding is happening a lot of times, so is it because the scan is called several times or something else?

changemenemo commented 3 years ago

Wouldn’t it be nice to have the actual same capabilities than Dropbox or others to be able to send several files or big loads of library? Or is it a limitation of the Android ecosystem? Because what’s the expected behavior of the user to add a nextcloud connection from its phone and upload his library? To have an empty library or a smaller library to begin with or the manual process of download the entire library and upload it himself through a computer to the nextcloud instance ?

CatEars commented 3 years ago

I recently installed nextcloud on my home server and I really love it so far. Tried the android app as well (version 3.14.1) and I was less impressed by that, mainly because of a hanging UI when there are a lot of auto-uploads. I think the core issue has to do with a large volume of files (as opposed to large individual files) but I have tried to read through the issues on github to see if anyone has pinpointed the issue yet, no luck. If the android app had a smooth experience like I had with the web and desktop interface, then nextcloud would surely be the best app of its kind!

Found this current issue thread that we are in and this similar issue https://github.com/nextcloud/android/issues/7378 so I assume that the problem is known.

I'd like to try and at least look into this issue so that it becomes clear what causes these crashes. @AndyScherzinger, where would the best place to report back any findings be? This issue, new one or another issue entirely?

CatEars commented 3 years ago

I used the following script to generate roughly 3000 files of 1-3 kB in size (so the total size is less than a phone recorded video):

import os

def random_bytes(N):
    return os.urandom(N)

for a in range(1, 5):
    for b in range(1, 10):
        for c in range(1, 10):
            for d in range(1, 10):
                fname = f'{a}{b}{c}{d}.txt'
                with open(fname, 'wb') as fhandle:
                    fhandle.write(random_bytes(a*b*c*d))

Then I ran latest version of the android app (ac29ae28e4cf13d558f87c4a03ba7cf8e7e3828d) against my home instance, running in docker with tag 20.0.4-apache and set up to use redis + mysql (following instructions on homepage to optimize nextcloud here: https://docs.nextcloud.com/server/16/admin_manual/installation/server_tuning.html). Redis and mysql also runs in docker with tags 6.0.9-alpine and docker.io/bitnami/mysql:8.0-debian-10 on the same host. I don't think redis or mysql is the issue here tough, just adding them for completeness sake.

I turned on auto-upload for the folder with the 3000 files and recorded the interaction and uploaded it to youtube, here https://www.youtube.com/watch?v=8-OYQ3_yEIM . This mimics quite closely what happens on my own phone. @boistordu, does this mimic your issues as well?

Now that I have managed to reproduce it in the emulator, I am going to see if there is some advanced profiler I can run that gives me better insight into what is actually going on under the hood.

CatEars commented 3 years ago

Looking at the log it seems to be writing QUERY _id < ? ... quite a lot, this is only triggered when every single ongoing upload operation is retrieved. Following that lead brought me to this: https://github.com/nextcloud/android/blob/master/src/main/java/com/owncloud/android/operations/UploadFileOperation.java#L375

        for (OCUpload ocUpload : uploadsStorageManager.getAllStoredUploads()) {
            if (ocUpload.getUploadId() == getOCUploadId()) {
                ocUpload.setFileSize(0);
                uploadsStorageManager.updateUpload(ocUpload);
                break;
            }
        }

I believe that each file that is uploaded is wrapped in an UploadFileOperation, thus when they all retrieve all ongoing uploads it kinda creates an O(N^2) behavior. I believe this is one source of the issue since this would cause a lot of overhead for each upload, making the startup of the app take exponentially longer time by the number of currently unsynced files.

I changed it to

        OCUpload storedUpload = uploadsStorageManager.getUploadById(getOCUploadId());
        if (storedUpload != null) {
            storedUpload.setFileSize(0);
            uploadsStorageManager.updateUpload(storedUpload);
        }

and tried it in the emulator. Logcat produces a lot fewer logs, but the UI is still unresponsive on startup. Gonna look a bit into where these operations are run from and how they force the UI to become unresponsive.

AndyScherzinger commented 3 years ago

They are run on the UI thread which is why the UI becomes unresponsive. With large numbers of files to. Make sure this works one would have to move all. Of it to a worker implementation 😔

CatEars commented 3 years ago

They are run on the UI thread which is why the UI becomes unresponsive. With large numbers of files to. Make sure this works one would have to move all. Of it to a worker implementation 😔

I managed to verify this as well. I commented out the following lines which start the file uploading, effectively stopping any form of file upload ever occuring and the app ran smoothly.

CatEars commented 3 years ago

@AndyScherzinger, would it be interesting to get a PR to solve this issue? Not sure exactly what needs to be done, but I have some time on my hands now over the holidays (and pandemic in general) and would be glad to help out.

AndyScherzinger commented 3 years ago

@CatEars YES, this would be awesome and very much appreciated ❤️ while I would suggest to start with one of the problem areas like upload existing files when activating auto upload on a folder.

CatEars commented 3 years ago

Thanks, Andy. I'll look into it =)

To anyone else passing through the ether and finding this issue, I managed to solve my issue on my personal phone somehow by ending up in a state where I was in the "Uploads" view (Hamburger menu > Uploads) and it was nicely and neatly uploading things one by one. I tricked the app by turning off and enabling wifi in different ways and somehow ended up in a non-hung state.

changemenemo commented 3 years ago

The last message is unclear but anyway I will wait for an update. I don't have enough time unfortunately and don't have any experience with Android code to actually help. Sorry. I didn't manage to make the collection of the logs in windows so... but andy anyway did already stated on the forum of nextcloud what was the problem, so I didn't need any other informations and let go of the problem in the end. Interface shouldn't be mixed with core function if you follow closely design patterns. But again since I don't anything about Android développement beside the fact that you need to follow their libraries...

CatEars commented 3 years ago

I have tried looking at a PR for this but couldn't really make it work. I looked into the recommended ways of doing background work, and the android documentation recommends two different ways. The WorkManager and the JobIntentService. The JobIntentService was very similar to the current implementation for uploading files (which uses a Service) so I decided to try that out. I got so far as to change the class and make it queue up the work, however I ran into bugs where the queued work would never run and I couldn't figure that out.

AndyScherzinger commented 3 years ago

I think for OS scheduled background work like auto uploads detected after activation we moved to WorkManager while @ezaquarii might know best ❤️

ezaquarii commented 3 years ago

@CatEars @AndyScherzinger @tobiasKaminsky

Yes, WorkManager is the official new API for this. The OS is in charge of launching tasks.

Recent Android versions have become more aggressive when it comes to power saving and do not allow applications to run things in background on their own, so WorkManager is pretty much the only viable gateway here.

We do have a modernized uploader/downloader service, but it is not used for syncing files. I had to pause my work on this due to other projects and family things taking my spare time, but I plan to resume it in January.

The challenge we have there is that various developers were "hacking" more and more overloads to the syncing API over time. You can start sync in so many ways, that porting it to a unified API turned to be a major development effort. I haven't found a way to do it in incremental fashion, so it's one hell of a rewrite and risk.

Quite frankly, I think we'll need some serious discussion with the stakeholders about it. I can't single-handedly make risky moves like that.

AndyScherzinger commented 3 years ago

I think this could still be split into two different efforts.

Solving the upload existing files issue described here
Agreeing on a general solution for handling client/server file upload/download operations as well. As any work that should be moved to the background (I am thinking of a far more advanced file operation queue (as in a DB table) which supports priorities as well as ordering in general and override entries by replacing them with new ones plus different characteristics when a certain queue entry is allowed to be processed (network, time, etc.))

github-actions[bot] commented 3 years ago

This bug report did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!

4ndr3aR commented 2 years ago

This still happens in June 2022, frankly the Nextcloud Android auto upload feature is mostly unusable for me now. It continues to re-upload the same 3800 files again and again. As soon as I get distracted and forget to press "wait" on screen hang, the app gets killed by Android.

luiandresgonzalez commented 2 years ago

Can confirm I am having this issue as well. I can provide any debug data that you require, but the app has been devouring resources from battery and phone processor since I activated an auto-upload from my moto One vision photo gallery.

nextcloud / android