Closed jochenwezel closed 10 years ago
RandolfCarter
commented
10 years ago just happened again with 1.4.1 :( Fortunately not with all files, but just with 33MB Unfortunately I can't provide a logfile of this time, and also the one from last time unfortunately seems to have gotten lost :(
lachlan-00
commented
10 years ago I think i've recreated this by renaming/changing files inside a folder causing the folder to redownload OTHER files in the same folder.
I have a folder where i dump my screenshots. Gnome saves the files with an invalid character (a colon ':') I used pyrenamer to get past the upload error and get them onto owncloud. Next sync after the upload the other files in that folder were downloaded again. but NOT the files that uploaded
It seems to be that when the contents of a folder are modified it causes a redownload of that folder. I'm waiting for this sync and a couple more to finish but i'll try and recreate this again and report back.
My log file is also 600mb just from those two syncs so it will take some time to parse it for those files.
Oliwally
commented
10 years ago I'm having the same prob. Windows Owncloud client continues to want to sync already existing files. Using client version 1.4.1 and 5.0.11 on my webhost (linux server). Got this happening on two computers.
I updated my system clock via the default time.windows.com, but problem persists.
I'm not particularly technically minded but happy to help out by providing log files if that helps :)
etiess
commented
10 years ago Hello @Oliwally
I think it will help the developer team if you could post your log. You can execute the following command on windows (in this example, log will be stored in the folder C:\test but you can change it if you want).
So you just have first to stop your sync client, and then launch the command:
C:\Program Files (x86)\ownCloud\owncloud.exe --logdir c:\test --logexpire 24
Then post the files here.
Thanks!
Oliwally
commented
10 years ago Thanks for the encouragement to chip in with useful info and for the instructions.
Sorry but the log was very long and I was unsure of how to post it here without it looking ridiculous. So here is a link to the file: https://turboworm.com/owncloud/public.php?service=files&t=31326c3d14c3e340b69aeeee5a56b1e1
Let me know if that link doesn't work - a bit experimental for me.
dragotin
commented
10 years ago @Oliwally in your case, you had no client database any more. In that case, the client redownloads and compares both versions byte wise. But you should not see that behaviour any more.
etiess
commented
10 years ago @dragotin Thank you for this analysis.
But why shouldn't @Oliwally see that behaviour any more? How does the client know that the files are the same now?
dragotin
commented
10 years ago I reproduced it on my computer. I have the encryption app enabled, and the result is exactly the same. Further debugging leads to owncloud/core#5090
@Oliwally do you also have the encryption app enabled?
torbennehmer
commented
10 years ago @dragotin: For the record: I have not been using encryption on the OC side at this point.
dragotin
commented
10 years ago @torbennehmer yes, the encryption thing seems only to be one problem here.
drewscm
commented
10 years ago Why when the desktop client update is installed does owncloud delete all files and re download????????
I did not mark the checkbox where is says delete contents.
Oliwally
commented
10 years ago Yes, I have the encryption app enabled but it's been enabled for a while now and the behaviour has just started. I'm guessing after the last client update.
Oliwally
commented
10 years ago Ok, so I've manually taken some of the larger files out of my owncloud storage (on the server and the local folder) to enable the sync to go a little faster because I was thinking that perhaps a part of the problem was trying to sync so many files (400MB). Now I'm down to about 30MB.
Problem now is that the Owncloud client crashes half way through. It's the same behaviour on two separate computers.
Here's the new log file for this: https://turboworm.com/owncloud/public.php?service=files&t=fba4ffde247598c3d91f83f4b23dda73
I then disabled the encryption on the server, logged out, back in and started the client up again, and although it re-synced the lot, it went through it all ok. It did create a whole bunch of 'conflict' files in my local folder (a duplicate of each file) and the sync complained about problems with individual files, but once I deleted those, the client behaved itself no probs.
Here's the log file for the last attempt: https://turboworm.com/owncloud/public.php?service=files&t=68c8496a1d92d0c2d0112a59766cee24
So at the moment it seems that encrypting is causing my probs. That's a shame - I'd really like to have my stuff encrypted.
Oliwally
commented
10 years ago I've now re-enabled encryption, logged out, logged back in, restarted the client and all seems fine. I've added a file to my local owncloud folder, had the client upload it - all sweet.
Latest log: https://turboworm.com/owncloud/public.php?service=files&t=24a8e0e502f9913c01d83522b1409925
Am I assuming correctly though that only files added after the re-enabling of encryption will actually be encrypted?
Oliwally
commented
10 years ago One problem I did encounter after disabling / enabling encryption is that my local files became unreadable (encrypted I guess). Couldn't figure out how to force the client to re-download the lot from the server, so I downloaded all files via the OC web interface manually and placed them into my local folder, replacing the encrypted existing files there. Worked, but it's a pain. But I'm getting a little off track here from the original problem, sorry.
fossxplorer
commented
10 years ago For your information: i don't use encryption at all, but still experiencing the issue.
RandolfCarter
commented
10 years ago @dragotin: this is definitely not only an encryption app problem. It just occured again for me with 1.4.1; I don't have (and never had) the encryption app enabled. I have now sent you the logfile, as requested above. Hope this can help resolve the issue.
etiess
commented
10 years ago It happened again @dragotin :-( My folder Photos was synced, and worked fine for a few days. And then owncloud crashed. I restarted the computer. And then the client started redownloading everything. The log is here: https://www.sugarsync.com/pf/D6476655_61894308_912595
Sorry guys, but I'm in a bad mood, disapointed.
In my opinion, OC is really not ready. I don't understand why you push new features (@karlitschek posted http://blog.karlitschek.de/2013/10/introducing-owncloud-6.html) while this product has so many fatal issues. You're dealing with user's files, OC has to be bullet proof (as "quickk" said there: http://forum.owncloud.org/viewtopic.php?f=23&t=9169). You present owncloud as an alternative of Dropbox, not as a beta version: the quality of OC is (very) far from Dropbox (and SugarSync, that I use daily for a year, without any problem!).
As a user, I think that quality should be the only improvement of OC6.
Today is a sad day for me: I cannot use OC anymore.
tenacioustechie
commented
10 years ago Hi, I'm experiencing the same issue described here. Server is 5.0.12, client is 1.4.0 and 1.4.1. Results to the earlier sql are below. Does this mean egat's are not being used on my setup? if so why not?
My setup does involve laptops only which are regularly disconnected, so I do have the suggested time drift between client and server. That combinded with no etags would create the problem I assume.
mysql> select count(_) from ocfilecache where etag = ''; +----------+ | count() | +----------+ | 1 | +----------+ 1 row in set (0.62 sec)
mysql> select count(_) from ocfilecache; +----------+ | count() | +----------+ | 556040 | +----------+ 1 row in set (0.00 sec)
Happy to provide more information, the re download is occurring regularly in my case, so I might be able to produce a log if people would still like more logs.
karlitschek
commented
10 years ago @etiess Please not that we are also working on fixing a lot of bugs and problems as mentioned in the blog post. About your problem: Just a speculation but it is possible that the crash of your machine destroyed the local sync database. In this case the only solution is to download all the files again to generate checksums and rebuild the local sync DB. This is the expected behaviour if the local DB is destroyed or removed.
obma
commented
10 years ago @karlitschek I would like to remark that to a loss or crash of the local sync DB it must not be the only solution to download all files again - that would be almost 200GB for me! It would save a lot of traffic and time to generate and compare hashes for files and folders before redownloading them.
etiess
commented
10 years ago @karlitschek : Your answer is a proof that OC is not bullet proof. The choice of a local DB based on etag cannot be reliable: crashes happen every day (windows, internet connection, owncloud client itself, server, ...). And in the actual configuration, OC doesn't seem to be able to deal with that: it's impossible to rebuilt the DB from scratch, without downloading everything. Not necessary to say that this "expected behaviour" is unthinkable (only in this thread, people speak about GBs of data!...).
zefrenchemen
commented
10 years ago Going through this as well. Blew over 30 Gb of data in the last week alone. It got so bad with 1.4.1 that I had to go back to 1.4.0. Even with that, still is an almost daily occurence... If I can assist in fixing this let me know. To me, this bug is absolutely critical : it has monetary implications as users will potentially go right past their monthly cap and occur extra charges (third time I'm getting a bill since using OC). This HAS to be stable to be used in production, I'm kinda losing faith here... Don't know if it's a clue, but this does seem to happen with the laptops only, I do not recall seeing this behavior on desktops... At least, not recently (did see it a while back, with older clients).
RandolfCarter
commented
10 years ago @dragotin regarding my logfile, seems I typed the email address wrongly. sent again now, hopefully it works this time and can help resolve the issue.
dragotin
commented
10 years ago @RandolfCarter yes, thanks, arrived here.
ways
commented
10 years ago I'm also experiencing this problem. All linux clients and linux server. All upgraded to latest versions from repository. All running ntpd.
dragotin
commented
10 years ago @RandolfCarter checked your logfile quickly and for me that looks quite like what is described in owncloud/core#5264. In your case it is the directory f...... in the sync folder top level.
torbennehmer
commented
10 years ago Hi dragotin,
I had the issue once again right this minute. The Client (1.4.1 on Win7 pro 64, against Server 5.0.12, Debian Wheezy) redownloaded one (and only one) of the two sync directories I have again. Interestingly, it did so right after a system restart immediately after the client startup. The client was launched manually a while after the system came up.
At that time, the system in question was the only one accessing my oc Sync files, the other three systems where both offline for a while.
I had the web interface open in a directory that was not part of the sync before. (I was checking a change I made in the second repository which did not redownload.) However, I did not change anything online, I just browsed the files there. Of course, when I opened OC, the top level directory (clientsync) which has been redownloaded, was visible once. No idea if it is important, I didn't access the web interface of OC for several days.
I have zipped up the following:
For the sample flie "Transit/Personalausweis.pdf" I have some additional information
access.log:217.94.130.114 - - [13/Oct/2013:14:32:54 +0200] "GET /remote.php/webdav/clientsync/Transit/Personalausweis.pdf HTTP/1.1" 200 620358 "-" "Mozilla/5.0 (Windows) csyncoC/0.90.2 neon/0.29.6"
As I am unsure, if there is more information you need, I also added todays access log.
It did also redownload the entire clientsync folder as did the first client.
Anyway, I added all client logs beginnin from the start of the second client up until it finished synchronizing everything also into the zip file. Maybe there is some correlation. The second system is running a current Ubuntu (13.04, 64 Bits) under XFCE (no Unity or Gnome)
As I am unsure, what sensitive information is involved here, please mail me at torben
https://owncloud.nehmer.net/public.php?service=files&t=93ed98527bc46d116b7e84798de82f70
I could also provide you with a current database dump along with tonights backup of the OC database. Please let me know if you think there could be anything of interest in there.
Oh and one thing: Contrary to some other commenters here I truly beleive that you are doing an amazing job with OC. I have my share of programming experience and building an enterprise grade file sync is anything but trivial. So as usual, please let me know if you need anything else.
Best regards Torben Nehmer
tenacioustechie
commented
10 years ago I've also got a log files that (I think) covers a re download of 13GB of files, Unfortunately we work over 3G mobile data, so you can imagine the bill. We are running Ubuntu 12.04 server, and Windows 7 and 8 clients.
A whole lot of files get compared like this in the log file: 10-11 17:48:29:021 _csync_detect_update: Database entry found, compare: 1290511333 <-> 1290511333, md5: 5253cd4c05b1c <-> 52327464b4a06, inode: 0 <-> 241223
Followed by a long list of these for each file in the directory. 10-11 17:48:30:912 _csync_merge_algorithm_visitor: INSTRUCTION_IGNORE file: ....doc 10-11 17:48:30:912 _csync_merge_algorithm_visitor: INSTRUCTION_NONE file: ....doc 10-11 17:48:30:912 _csync_merge_algorithm_visitor: INSTRUCTION_SYNC
Happy to provide the log file to someone directly if it would help (its 192Mb)
moscicki
commented
10 years ago Hello,
I have the same issue: the client started downloading now everything from the server (many GBs over a wireless abroad) - even though nothing has changed.
So there is something that invalidated etags on the server-side - I guess that somehow is related to a mixup in the web-access layer that triggered some core function to recreate the etags.
@karlitschek: what about enabling saving audit information whenever the etag is modified on the server so that eventually we can trap the culprit and see the call stack? Unless you already have an idea how to fix that, I would see this as an opportunity for the developers to get meaningful reports for cases like this. This could be just a simple add-on patch to do more verbose logging (xdebug style) for this particular code point. I guess people affected by this problem would be willing to add it onto their current server setups to help you out debugging.
kuba
On Oct 14, 2013, at 3:21 PM, Brian Mills notifications@github.com wrote:
I've also got a log files that (I think) covers a re download of 13GB of files, Unfortunately we work over 3G mobile data, so you can imagine the bill. We are running Ubuntu 12.04 server, and Windows 7 and 8 clients.
A whole lot of files get compared like this in the log file: 10-11 17:48:29:021 csyncdetect_update: Database entry found, compare: 1290511333 <-> 1290511333, md5: 5253cd4c05b1c <-> 52327464b4a06, inode: 0 <-> 241223
Followed by a long list of these for each file in the directory. 10-11 17:48:30:912 csyncmerge_algorithm_visitor: INSTRUCTION_IGNORE file: ....doc 10-11 17:48:30:912 csyncmerge_algorithm_visitor: INSTRUCTION_NONE file: ....doc 10-11 17:48:30:912 csyncmerge_algorithm_visitor: INSTRUCTION_SYNC
Happy to provide the log file to someone directly if it would help (its 192Mb)
— Reply to this email directly or view it on GitHub.
drewscm
commented
10 years ago I switched to a different windows user and it cause Owncloud to completely re download my files.
Does anyone know a fix or stop owncloud when a different user logs on?
drewscm
commented
10 years ago It's done so good for a while... Now my files are screwed up now. I have never had these issues with Dropbox, I think I am going to have to give up on Owncloud until they are able to develop into a stable application. They should put in bold letters to backup all files.... Luckily that's what I did everyday! Otherwise I would be sitting in my office twiddling my thumbs because all of my files vanished, and have to wait for a gig to download at 1.4 mbps
These types of issues should be at the top of any bug lost before any convience items. Until the re downloading and deleting is fixed this program is not stable! Very upsetting
dragotin
commented
10 years ago @drewscm It would help if you could open a new bugreport following https://github.com/owncloud/mirall/blob/master/CONTRIBUTING.md and describe what happened so that we can fix it. Sorry for the trouble.
fossxplorer
commented
10 years ago @dragotin @danimo What do you really need of info to try to fix this one? IMO, such an issue is a showstopper for the whole ownCloud!
zefrenchemen
commented
10 years ago A client just called me with this problem : he described precisely what he did prior to the problem occuring, hope it can help. He was looking for a file in a shared folder. Could not find it locally has he realized his "Shared" folder was not synced by his client. He went to the web interface, found the file, downloaded, edited and re-uploaded the single file. A few seconds later, the client on his machine started to download ALL of his synced contents (+20Gb), although the file he edited was not in said synced folders... Hope this timeline helps. Things are becoming rocky here with this problem, update on a potential fix would be awesome. Can anyone confirm in the meantime that downgrading to pre-1.4.x would eliminate this issue for the time being ?
@dragotin , as aggravating as this problem is, I appreciate all the hard work you and the other devs are putting in. I'm no programmer, but I'm willing to assist any way I can.
protree
commented
10 years ago Hi all, I have the same issue and was able to fix it temporary by resetting the sync journal as described here:
"Pressing F5 in the Account Settings Dialog that allows to “reset” the journal. That can be used to recreate the journal database. Use this only if advised to do so by the developer or support staff." (http://doc.owncloud.org/desktop/1.4/architecture.html#the-sync-journal.)
Honestly, I don't know if this is a good idea and it doesn't seem to fix the root of the problem, because after some days it started to resync again but this might be a quick fix until something is done. I was even able to stop a running resync with this method and lost no data so far, but as I said I don't know if this is a good idea though (read the advise in the quote above)
Oliwally
commented
10 years ago Don't know if this helps at all but I've had no more problems to date after my last post (just over 2 weeks ago). All seems to be running sweet. I did end up re-downloading everything at that stage which was ok for me - I didn't have much in my owncloud folders.
mdieterich
commented
10 years ago I read over this bug report earlier today, while looking for something else, only to have it happen to me about 15 minutes ago... I think it's contagious! I got bit with downloading 13Gb of extraneous data, after all I did was add a folder to be shared with someone else and adjusting my share settings. Unfortunately, I didn't have client logging on at the time, but I do now.
If it matters, I'm running OC 5.0.12, with client version 1.4.1 under Linux. The machine is always on and running ntpd, so my experience would add to the voices saying it's not a time sync issue. I'll let the group know if I discover anything else.
Mark
BenS89
commented
10 years ago It wasn´t until the release of version 1.4.1 that this issue occurred. I´m using Server 5.0.12 and client 1.4.2. Nearly everyday the client syncs the whole root folder even though there are absolutely no changes. That´s why we had to delete the sync client. In my opinion, this issue should be rated with highest priority.
nileshgr
commented
10 years ago I'm waiting for this issue to be closed and actually downgraded to 1.3 long ago when 1.4 was just released. On Oct 26, 2013 12:16 AM, "BenS89" notifications@github.com wrote:
It wasn´t until the release of version 1.4.1 that this issue occurred. I´m using Server 5.0.12 and client 1.4.2. Nearly everyday the client syncs the whole root folder even though there are absolutely no changes. That´s why we had to delete the sync client. In my opinion, this issue should be rated with highest priority.
— Reply to this email directly or view it on GitHubhttps://github.com/owncloud/mirall/issues/994#issuecomment-27116412 .
dragotin
commented
10 years ago We know that this is a server issue. It is tracked in owncloud/core#5264
Be assured that we are working with highest priority to get that fixed. But it's at tricky one. Please, if you have time, read through the mentioned core bug, maybe you have a useful hint.
nileshgr
commented
10 years ago I'm not sure if this is a server issue or not because I'm using 1.3 version of the client with latest server and it's totally fine.
Unless there are some changes in 1.4 to make it more compatible with the server... I'm unaware if any. On Oct 26, 2013 12:27 AM, "dragotin" notifications@github.com wrote:
We know that this is a server issue. It is tracked in owncloud/core#5264https://github.com/owncloud/core/issues/5264
Be assured that we are working with highest priority to get that fixed. But it's at tricky one. Please, if you have time, read through the mentioned core bug, maybe you have a useful hint.
— Reply to this email directly or view it on GitHubhttps://github.com/owncloud/mirall/issues/994#issuecomment-27117146 .
BenS89
commented
10 years ago I can not prove it but I am relatively sure that this issue occurred the first time with client 1.4. I tried all other versions too.
dragotin
commented
10 years ago In all logfiles I saw so far it shows that the download starts because there are different etags reported from the server. ETags only change on any kind of write operation on files. These could be identified in the server access_log. Do you see any unexpected PUT or PROPPATCH requests with the 1.4 clients that you do not see with earlier releases?
BenS89
commented
10 years ago Maybe another hint: If I use just one client, for example the windows client on a Win 7 laptop, there are no problems. As soon as I use my private Mac after using the Win7 client, the Mac starts syncing the whole directory. When I continue to use the Mac client, everythin works fine. When I switch to the win client again after using the Mac client, the Win7 client syncs the whole directory.
Maybe, the changing etags could occur when using different plattforms with different clients? Just an idea..
h4knet
commented
10 years ago Hello, I am just leaving a message because i really often get the same problem (downloading a whole folder again and again). The server is running 5.0.12 on a debian linux with Apache2,mysql,php5. The clients are on Windows 7 Pro N 64bit and Windows 8 Pro 64 bits 1.4.2.
I hope this bug will be fixed soon because it's really embarassing and you can't work efficiently with this bug always let you wasting time to wait syncing your whole owncloud folders.
drewscm
commented
10 years ago It's not just different platforms, I had the issue with two win 7 pro 64 systems. The last time it happened, it really screwed things up bad. The first thing I did was log into my server and deleted my accounts and shut it down.... Then I opened my Dropbox up. I was spending more time fixing things than using it as a tool. I would like to help, because I love the idea of controlling my own data, but I just don't have the time right now. I will go back in a few months after a couple more versions.
mkay
commented
10 years ago +1 here setup is debian wheezy/apache2 on a dedicated server running 5.0.12 and two 1.4.2 OSX clients on the same LAN. Random re-downloads every now and then. Happy to provide logs if needed. Sadly this makes the whole tool unusable.
luksam
commented
10 years ago Hello, I use only one client 1.4.2 (on windows 8) connected to a server 5.0.11. => Reproduced also with client 1.3.0
In order to help, I have client log 1.4.2 with the issue: http://dl.free.fr/keOPJd0Bf owncloud.log.5: log of the normal upload of files on the server owncloud.log.9: log of "unnecessarily download" of files on the client (download without any user actions on the server and the client (no new file or updated file, repository renamed ..)
Analysing log, it doesn't seem that etag of downloaded files had changed on the server between the upload and the download ...
Regards,
masterblaster79
commented
10 years ago Hi guys.
Unfortunately I got the same issues with client 1.4.2 on Fedora and Server 5.0.12, here are the relevant parts from my client log file:
A few seconds later it goes through the same loop...
Inode on server side is not 0 but 983730 root@xxxx:/var/log/mysql# ls -i /var/www/owncloud/data/stephan/files/some-file.doc 983730 /var/www/owncloud/data/stephan/files/some-file.doc
And it looks like owncloud compares with the inode on client side (differs from the one is the log above at file has been overwritten again in the mantime)??? [stephan@client ~]$ ls -i ownCloud/some-file.doc 2359666 ownCloud/some-file.doc
On server side the etag is: 51a20757a65c6 On client side the etag is: 51a20757a65c6-gzip
I hope this helps, I can provide complete log files on request.
Cheers! Stephan
A user shared with me a folder with about 10 10 GB data (approx. 200 files).
I don't know why, but it's often (not always) finding some new files. Download follows - but unnecessarily: files are absolutely identical (byte comparison); the file on server hasn't been touched for months. So the file hasn't changed.
Following suggestions:
Local client: 1.4.0 on Win2008 R2 server (and other win systems, too) Server version: 4.5.7 on Linux