Owncloud deletes files & tries to sync again (with little success)

Phlos commented 6 years ago

I have a 250 Gb storage allowance through a university system which I use to sync my files between several computers that I work on. The university system supplies a tailored client for windows, but for linux I use OwnCloud. I noticed in the last few days that OwnCloud is deleting files from a local computer and then tries to redownload them. This is an issue I have had several times in the past and somehow keeps popping up. [edit: That it started happening again is possibly related to the fact that I updated to 2.4.2 on 19 July).] I have only experienced this on my linux machines, never with Windows (although it should be noted that I sync only a rather small subset of my stuff to the Windows OS).

Expected behaviour

I have the OwnCloud client on my laptop. It syncs my files. If I am offline for a while, it syncs everything as soon as I'm back online.

Actual behaviour

I have the OwnCloud client on my laptop. It deletes files that I had on the local machine and on the server from the local machine, then tries to redownload them.

The sync for redownloading is not unproblematic:

at some point, it stops downloading and loses the connection, and then sits stuck -- as far as I can see, until I forcefully do something about it. The connection issue is not solved by clicking log out and log in alone (as far as I can see); I have to restart owncloud and then it connects.
at each of the retries, it tells me for a different set of directories that "a new directory > 500MB has been discovered" which I should approve the download of (500 MB is the prompt limit in my client).
This is especially problematic because in general (also when OC is not deleting files) it takes ages to initialise OC after a restart - it takes absolutely forever to work through all the directories and files to check whether any changes have taken place.
Owncloud has, up until now, not realised the new folders that I have created during my time offline.

Consequently, I wait for hours to get a sync that maybe 30 minutes, after which it stops working, I restart everything, etc.

Steps to reproduce

Since this is not a very straightforward issue, I'm afraid there are no straightforward steps either. I did the following, though:

Have owncloud on my laptop (Ubuntu 16.04; currently I have OC 2.4.2)
Work on it without internet for a while.
Connect back to the internet
Discover the file & directory deletion mayhem.

Server configuration

Unfortunately, I do not know what specs the university system has.

Client configuration

Client version: 2.4.2

Operating system: Ubuntu 16.04

OS language: English (United States)

Qt version used by client package (Linux only, see also Settings dialog): Not sure, but the settings dialog says "Built from Git revision d6e975 on Jul 19 10:54:56 using Qt 5.6.2 OpenSSL 1.0.2g 1 Mar 2016"

Client package (From ownCloud or distro) (Linux only): I'm not sure what is meant by this question.

Installation path of client: /opt/ownCloud/

Logs

I had no logs up until now -- using the instructions below I have now started logging what is going on. I'll leave my laptop at work and connected overnight, and come back tomorrow to see what the dawn brings...

Other comments:

This issue might be related to #3102, #6282 and #6322, but since my issue seems to be subtly different (and since I don't understand half of those conversations), I hesitantly file this as a new issue. Apologies if this is not the way forward.

ckamm commented 6 years ago

@ogoffart I don't see suspicous log entries. :(

At 08-18 22:28:54:242 it successfully reads 406315 entries below "work"
During the local discovery there are thousands of "Database entry found" log messages for files inside "work/" as well as for the folder itself. Spot check: the file "work/website/doc/Makefile" which is deleted later was mentioned as "Database entry found" during the local discovery.
At 08-19 00:29:26:039 just a couple of ms before failing to read the "work" data, it logs "1 entries read below path todo from db", which is the correct number of entries for the "todo" directory.

ckamm commented 6 years ago

Note, at 08-18 16:25:5 a sync failed with the error "Free space on disk is less than 48 MB". Maybe this somehow influenced the db? Even if sync finished successfully several times afterwards.

Phlos commented 6 years ago

@ckamm I doubt it, because the disk space issue was only relevant this day. All the other times files got deleted, there was plenty of space. (on 08-18, I had temporarily rsynced a rather large subdirectory to my homedir, i.e. where it was safe from deletion ;) )

tomneedham commented 6 years ago

Interesting before the 05:26 deletion, there are two PROPFINDS from the server:

<CLIENT IP> - - [17/Aug/2018:05:25:23 +0200] "PROPFIND /files/remote.php/dav/files/<USERNAME>/ HTTP/1.1" 207 501 "-" "Mozilla/5.0 (Linux) mirall/2.4.3 (build 10035)" microsecs:129403 response_size:501 bytes_received:800 bytes_sent:1284 W3ZAI1HyHwQdPnjhy9YCiwAAACE
<CLIENT IP> - - [17/Aug/2018:05:25:34 +0200] "PROPFIND /files/remote.php/dav/files/<USERNAME>/ HTTP/1.1" 207 14829 "-" "Mozilla/5.0 (Linux) mirall/2.4.3 (build 10035)" microsecs:396194 response_size:14829 bytes_received:1049 bytes_sent:15640 W3ZALtbL2ItCwCIiNDF4bQAAAAc

All the previous PROPFINDS have the same size, but the last one before the deletion has quite a larger size in the response. 14829 vs 501 bytes. This could lead us down the thought path that it is an incorrect server response that is triggering the client to delete the files. mitmproxy with the request/response bodies here would be what we need next - but how do we capture just this, at the right moment?

T0mWz commented 6 years ago

Was still thinking about this issue. Was wondering if there were changes on your side @Phlos . Will try to make a change on our site for some webservers, to see of the strange 401 PROPFIND response can be fixed.

@Owncloud Guys; Currently we capture 401 response to another page due to unauthorized user logins. I will remove this 401 redirect for the oAuth2 sync client webservers, too see or that may have an impact on the strange propfind result of this specific issue. But would expect that the client would handle this properly and would not show such as a strange behavior with propfind / locale remove action.

@Phlos, would you perhaps be able to try again to see of it makes a difference. Super thanks for your help!

Phlos commented 6 years ago

@T0mWz sorry for the late reaction! I haven't had any more deleterious experiences recently. I strongly pruned my data, though, and only have subsets of my directory tree synced to my laptop, not the whole thing. At least for now, it seems to work, but I'm holding my breath continuously... ;)

Could there be a relation with directory size / number of files / depth/breadth of tree?

Phlos commented 6 years ago

Something weird just happened again as I was looking at it. Unfortunately, no logging took place, but it was the directory work/EMed/plots that got deleted -- seemingly on the server side, because it still existed on my laptop. Now it is uploading the directory from my local laptop dir (where it was retained). VERY strange.....

@T0mWz can you see anything in the server logs around 15:19:16?

ckamm commented 6 years ago

@Phlos About logging: If you're using 2.5.0 definitely

open the client settings
hit F12, the log window should pop up
enable "Permanently save logs"
hover over the checkbox and take note of the directory logs will be stored in

That way detailed logs will be written all the time and you'll have a couple of hours to make a copy of them when something strange-looking happens. You only need to do this once - it's a new feature to make it easier to have logs for rare, hard-to-reproduce issues like this.

ogoffart commented 6 years ago

I opened https://github.com/owncloud/client/issues/6814 to track the oauth2 and timeout issue.

(So far no progress have been done for a month regarding the reason why the directories got deleted)

Phlos commented 6 years ago

Update: after purging my tree, for a while the problem didn't show up. However, it had been growing again since, and last week (of course just before I wanted to take the laptop along with me on a trip) it decided to do the big ol' delete again.

Potentially having a big tree not only results in the one hour discovery cutoff (which inhibits re-download, as per #6814 ) but also somehow results in deletion. Related?

@ckamm I tried the F12 option, but unfortunately the owncloud log dir defaults to my /tmp/ which is in my root which is extremely full as it is already - will have to start it up manually. Maybe something to change, i.e. let the user decide on the log dir location?

I'll now and tune down my selective sync in hopes of building up the tree again..

T0mWz commented 6 years ago

Hi @Phlos ,

Painful to hear. :(

During the upgrade begin of October, we upgrade to latest OC version and latest oAuth version, assuming that this could possibly help in combination with latest OC sync-client. However, this has reset the AccessToken lifetime back to the default 1 hour.

Will restore this again to a longer period, so hopefully the sync client has some more breath to complete a sync. However .. this can not be infinite and would like to start with an increase to 2 hours.

@ogoffart, I think it's a hard subject. But is there a way that a client can better deal with an expired oAuth AccessToken?

michaelstingl commented 6 years ago

But is there a way that a client can better deal with an expired oAuth AccessToken?

@T0mWz Solution in https://github.com/owncloud/client/issues/6814 and https://github.com/owncloud/client/pull/6819 . You could test with the latest daily builds from https://download.owncloud.com/desktop/daily/?C=M;O=D

Phlos commented 5 years ago

It's happened again. Near-complete deletion of my largest directory w/ most elaborate tree.

I was working on my laptop yesterday after not having used it for about a week. It then proceeded to download most of the stuff in this one directory while I was working in another. This other directory keeps syncing fine (have in between worked on another machine on the same files). [this is an adapted message because I thought that the deletion had taken place today] [Original message:] ~~I was working on the laptop yesterday early afternoon, all fine - hibernated, continued working on a desktop, all fine - started working again on the laptop, directory where I was working was behaving normally,but I just noticed that it had deleted everything in this other directory. Could hibernation have anything to do with it? ~~

I've got logs, if anyone's interested... owncloud_deletion

ckamm commented 5 years ago

@Phlos Logs are very interesting! mail at ckamm de.

ckamm commented 5 years ago

The issue I see in the log is similar to what was observed before:

// local discovery example (thousands of entries)
02-28 09:11:12:885 [ info sync.csync.updater ]: Database entry found, compare: 1467030292 <-> 1467030292, etag:  <-> 3e981534e5b53cfc77ece8d17fa05013, inode: 2778272 <-> 2778272, size: 4096 <-> 4096, perms: 0 <-> 9f, type: 0 <-> 0, checksum:  <-> SHA1:, ignore: 0
02-28 09:11:12:885 [ info sync.csync.updater ]: file: A/B/C/file, instruction: INSTRUCTION_NONE 

// remote discovery doesn't find a single one
02-28 09:11:17:404 [ info sync.csync.updater ]: Reading from database: A/B
02-28 09:11:17:404 [ info sync.csync.updater ]: file: A/B, instruction: INSTRUCTION_NONE <<=
02-28 09:11:26:396 [ info sync.csync.updater ]: 0 entries read below path A/B from db.

I'll continue to look for potential causes.

ckamm commented 5 years ago

The issue I see is that SqlQuery::next() doesn't distinguish between end-of-data and errors. It's possible that it returns false due to an error and the function that reads from db assumes there simply isn't any data. I'll fix that.

guruz commented 5 years ago

Ouch! Thanks a lot @ckamm for fixing. @Phlos @T0mWz thanks for the logs and patience..

jnweiger commented 5 years ago

not tested with 2.5.4

It appears hard to reproduce a setup locally, that would trigger the issue. Not tested here. @Phlos can probably only retest after we have a released branded 2.5.4 client. -> suggest 'not tested' is acceptable for proceeding with 2.5.4rc1

Okayish

zinalili commented 5 years ago

It just happened to me on Windows 2.5.4. A folder was still syncing, I renamed it and all the other folders got deleted. Only this one left. Hopping on beta channel now, hoping for the better.

owncloud / client