Let synchronization be smarter / faster

[Aesculapius]

I have set up an owncloud account with quite some data and folder structure in it. Obviously, syncing that would take some time so it isn't strange that not everything is visible at my phone right at once.

But it seems that the client is trying to sync everything from A-Z and in that order, while I'd think that syncing that what I need the most would be preferable. And what I need the most, are these items that I'm actually looking for. So if I browse into a folder "Work", it can be assumed that I want to see its contents, so syncing the rest should be stopped and the contents of the Work-folder should be shown.

This way, by prioritizing the syncing, the user experience is way more natural, since you don't notice that some folders haven't been synced yet (as you haven't watched in them yet). When you compare owncloud to dropbox (which works this way), there is a significant difference in the time it takes to show a directory.

Actually, thinking about it. On the android client, which behaves a lot like the dropbox client on android right?, the server is 'always right', why would syncing a file list then take so long anyway?

[davivel]

Good point.

We have some imprevements pending to apply in synchronization, some of them to make it faster. But this one would be really pro.

[tripflex]

Yeah i'm having this same problem as well, it seems the refresh does check every folder and having a large folder structure ends up having the client run on my phone all day and kill my battery about 5x as fast as it normally does.

I was going to actually look at trying to setup a fix just to have an option in settings to refresh just a folder you're looking at instead of the entire structure. Right now if you click settings and then refresh it will refresh everything which makes the app pretty much un-usable as it takes over a day just to update.

[davivel]

Yes, currently the refresh is only for the full account. At this moment I only can ask you to be patient. Not trying to avoid the issue, we really have some important tweaks pending to apply for the synchronization, but right now we have to finish some other features first.

[Aesculapius]

davivel, great to hear about. Is there any roadmap or eta for this? Then we'll patiently wait for it ;-)

[davivel]

I know patience is easier with a roadmap, but there is no real timeline for this.

Yes, we ask a lot of patience :) . Not trying to be cruel, we appreciate a lot the interest of everybody in every single issue, but there is a lot of things to do and priorities sometimes change faster than we can react.

The only I can promise is that I will notice it in this issue as soon as we have some advance in it.

[tripflex]

Thanks davivel, in the meantime would it be difficult to add a menu item to just refresh that folder?

With the size of my cloud now if you use refresh on the android client it won't be done syncing until about 12-24 hours later...which completely kills the battery and almost makes it a show stopper .. or maybe even the ability to select folders to not sync would help...not sure. Unfortunately I know nothing about android dev otherwise I would help :(

[a-schild]

See the same problems with ~20200 files in hundreds subfolders.

Related issues: #121 and #124

[butonic]

Related #207

[benedetto]

I have to admit that I have not the smallest clue of how the syncing is done or what happens on the communication layer between the app and the server. However, if I understand it correctly, at the moment the app is parsing through the directory tree (how? by looking at the webdav files?). Due to poor performance of mobile CPUs and a maybe bad network bandwidth this can take ages for bigger directory trees with lots of sublevels.

Wouldn't it be easier to let the owncloud server – which has a lot more computation power and already should keep a database of all the files – calculate all the needed information, so that the app only has to fetch a file containing all the information or ask for "give me the actual directory structure".

Just my 2 cents, if this is total nonsense, just let me know...

[butonic]

Actually, webdav provides a so called etag for every file and folder that changes whenever the item is updated. A folders etag will change when a file inside it is changed. This allows a sync client to just look at / and compate the etag to the etag remembered from the last sync. If they are equal nothing has changed. If not the client needs to compate the etags of the children of / or the current folder in the same manner.

I think this is the way mirall syncs. @danimo @dragotin correct me if I'm wrong or just verify this with a "correct" statement please

The only thing I am unsure of is wether a directory listing / getting a collection in webdav lingua returns the etag of all the elements in it, which would be necessary to make this performant.

benedetto notifications@github.com schrieb:

I have to admit that I have not the smallest clue of how the syncing is done or what happens on the communication layer between the app and the server. However, if I understand it correctly, at the moment the app is parsing through the directory tree (how? by looking at the webdav files?). Due to poor performance of mobile CPUs and a maybe bad network bandwidth this can take ages for bigger directory trees with lots of sublevels.

Wouldn't it be easier to let the owncloud server – which has a lot more computation power and already should keep a database of all the files – calculate all the needed information, so that the app only has to fetch a file containing all the information or ask for "give me the actual directory structure".

Just my 2 cents, if this is total nonsense, just let me know...

---

Reply to this email directly or view it on GitHub: https://github.com/owncloud/android/issues/96#issuecomment-20513705

Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

[davivel]

The etag is received through the WebDAV interface with no problem. The client app is almost ready to use it, but we need to find the right time slot to finish the implementation.

@benedetto , your proposal would imply the definition of a new interface for synchronization between server and client. There are a lot of improvements we can make using WebDAV.

[a-schild]

Using the etag can probably solve the sync for later sync attempts. But I don't think it will help in the initial sync which takes a few hours when you have a large tree (20'000 or more files) And since the mobile device often are not online that long, the initial sync will restart and restart and restart....

[dragotin]

Hmm, why don't you just work with csync, the library we're using for syncing in mirall, the ownCloud client? csync is written in C and does everything for you when it comes to syncing. We are maintaining it and it would make big sense to use it on all platforms.

[karlitschek]

I agree @dragotin. Sharing the code (csync) is strongly recommended. A lot of thought went into csync already so it would be bad to not use it here.

[butonic]

sure, but ... um, the app, like nearly all android apps, is written in java and we would need to either write a wrapper or use the native jdk, something few people are familiar with.

[butonic]

To give you an idea of what would be necessary: Using NDK to Call C code from Android Apps

made to break ... IMHO

[danimo]

Not seeing anything particular dangerous here, but maybe I've seen too much JNI/NDK stuff in Qt for Android.

[dragotin]

On 05.07.2013 15:40, Daniel Molkentin wrote:

Not seeing anything particular dangerous here, but maybe I've seen too much JNI/NDK stuff in Qt for Android. Maybe Qt for Android is an option in general?

Qt 5.1 was just released with tech previews for both Android and iOS: http://developers.slashdot.org/story/13/07/04/2127227/digia-releases-qt-51-with-preliminary-support-for-android-and-ios

That would bring all clients (!) on one platform. Sounds appealing imo.

[maystar]

For me this issue is really important, since my initial synchronization never succeeds. After a few minutes I always got the failure message. Some new files and folders appear afterwards anyway, but not all. Thus the client is useless for me. My account contains a big amount of music files and pictures and my phone is a Sony Xperia Neo V, which is not the most powerful on earth. So I guess it's a memory problem. But I agree to benedetto: What are the advantages for the client to scan the files by itself over webdav? Why isn't the client just getting the information from the server, which scans the files anyway?? Assume a system with 1000 user, each with a tablet and a phone periodically syncing with mobile clients. And all users share a common directory with many files. If this files now changes in a very short time period the files are scanned 2000 times instead of just once by the server! I think in that case not only the mobile clients get a problem.

[davivel]

I don't know how difficult would or wouldn't be using Qt for the app. But probably we would be talking about restarting the app from scratch.

About csync, if you could provide us some documentation about the way it works, specially what are their dependencies, we could make some estimation about the possibility of integrate it as a native extension.

About synchronization as an overall: we have A LOT of ideas about how to improve it, without the need of dropping the WebDAV interface or changing the app architecture. We can recover and add them to the issue tracker if you like.

[blurmonk]

Sorry to bug this thread. I am still trying to understand this syc thing under Android. When the client syncs, what is it syncing? I see no file syncing on the surface, so why is it trying to sync stuff? Is syncing referring some other thing or is it a real syncing as we understand (mirroring). The reason I am all confused is that OC Android client seems to use alot of cpu on my 4.2.2 (therefore I am looking at this bug), but I see no files downloaded to the owncloud folder . So what is it that is causing high cpu usage and battery drain?

[davivel]

" When the client syncs, what is it syncing?" The folder tree for the full account, including names, size an type of every single files. It also synchronizes the content of files with the option 'Keep synchonized" checked, but not for any file with the options unchecked, what is the default value.

"I see no file syncing on the surface, so why is it trying to sync stuff?" To let the user have something to browse.

"Is syncing referring some other thing or is it a real syncing as we understand (mirroring)" Sorry, but we can not know what every single people understand with 'syncing'. Even though, we never used the term 'mirroring'.

"So what is it that is causing high cpu usage and battery drain?" We cannot guess it. We would need more information about the way you use the app. How many accounts do you have in the app? Do you often change from one to other? Does the synchronization process finish without error? Do you have files checked to be 'kept-in-sync'? How many files and folders are there in your accounts? What is the model of your device? Did you measure the time your battery is alife with and without the app installed?

[blurmonk]

davivel,

Thanks for your reply. That clears up the issue for me. As I see it this sync is per file on demand type of sync. It could probably help to make that clear in the account page. Because when one ticks the sync box, he/she generally assumes that it means the whole account. At least that was the case for me.

On the other hand it would be much more useful if syninc is actually per folder rather than per file. File numbers in a folder might change. So when a user can keep a folder in sync that way one does not need to tick un tick many files.

In any case thanks for the greta work.

[davivel]

@blurmonk , thanks to you for your comments. We know there are a lot of people interested in some improvements in synchronization, such as the sync per folder you comments, or the update of the list of files in a folder with just browsing. All these things are commented in different issues in the repository.

We are trying very hard to find the right time slot to work in all these. We cannot provide a deadline for this, but we really hope we can take it as highest priority very soon.

[tripflex]

And as always, i'm sure I speak for everyone else when I say we appreciate all the work you guys have put into this as we all know time is money. Thanks again!

[neebski]

+1 Current folder refresh vs entire account

[davivel]

Hi everybody here.

There is a lot of good ideas in this topic, and one of the most common is 'refreshing first the current folder'. That is already achieved in version 1.5.0, already released. Let me invite you all to try it.

For once, I will close this issue without waiting your confirmation. There are some other good ideas here, and we have some others too pending to include in the repository. Please, create * new issues * for those ideas if you feel they need to be open. Mixing too many things in the same issue makes more difficult for as tracking what's already done, on what isn't.

Thanks everybody

[ghost]

As this issue now closed, someone can claim the bounty at https://www.bountysource.com/issues/541709-let-synchronization-be-smarter-faster - it's just 5$, but still...

[davivel]

Really? Who posted it there? XD XD XD

[ccverg]

@davivel it was an anonymous backer, but if you closed the underlying issue here (refreshing first the current folder) you should go claim the bounty!