client 1.4 unnecessarily downloads again those files which are already downloaded

jochenwezel commented 11 years ago

A user shared with me a folder with about 10 10 GB data (approx. 200 files).

I don't know why, but it's often (not always) finding some new files. Download follows - but unnecessarily: files are absolutely identical (byte comparison); the file on server hasn't been touched for months. So the file hasn't changed.

Following suggestions:

Finding and fixing the reason why the client trys to re-download equal files which hasn't been changed/updated in any way
Before downloading files: download a hash of the file from server for that file + check if hash can be found in local journal database + if yes: copy the file locally

Local client: 1.4.0 on Win2008 R2 server (and other win systems, too) Server version: 4.5.7 on Linux

dragotin commented 10 years ago

Its obvious that the etag should not be end on -gzip

masterblaster79 commented 10 years ago

Hello dragotin.

Its obvious that the etag should not be end on -gzip

But that's what the client has in the DB, no idea where this comes from. Looks like this etag mismatch is the problem here.

jtessier-oriso apparently has the same issue (see above).

I am using these owncloud client from repo isv_ownCloud_devel [stephan@nas ~]$ rpm -qa | egrep owncloud|ocsync libocsync-plugin-owncloud-0.90.4-1.1.x86_64 libocsync0-0.90.4-1.1.x86_64 owncloud-client-1.4.2-1.3.x86_64 [stephan@nas ~]$ LC_ALL=C yum info $(rpm -qa | egrep owncloud|ocsync) | grep "From repo" From repo : isv_ownCloud_devel From repo : isv_ownCloud_devel From repo : isv_ownCloud_devel

Server is on Wheezy but has Apache 2.6 / mod_fcgid

Edit: Client is on Fedora 19

Unfortunately I have no clue what causes this :-(

dragotin commented 10 years ago

@masterblaster79 can you check in the server database, table oc_filecache? It's interesting if the etag entry in the database has also the -gip.

This sql statement:

mysql> select path, etag from oc_filecache where path like '%filename%';

replace filename by the name of a failing file ;-)

masterblaster79 commented 10 years ago

Hi dragotin.

There is no "-gzip" in the server DB.

Why does the file appear twice? I checked some other files as well, some appear twice, some not.

Server location of the file is: /var/www/owncloud/data/stephan/files/super8-2.m4v

In the log file the etag is 51a20757a65c6-gzip (which apart from the -gzip matches the first result).

Regards, Stephan

dragotin commented 10 years ago

@icewind1991 do you have suggestions for us here?

icetear commented 10 years ago

Hello,

I am experiencing the same problem. And I just wanted to contribute some log lines that might be of interest. I don't know if this is related, as I don't know the meaning of the error messages.

11-11 21:34:27:534 csync_ftw: Uniq ID from Database: Basilisk Games/Book 2 Saved Games/slot2/5051.ent -> 11-11 21:34:27:534 csync_walker: file: /Users/mario/owncloud-icepalace/Basilisk Games/Book 2 Saved Games/slot2/5051.ent 11-11 21:34:27:534 _csync_detect_update: ==> file: Basilisk Games/Book 2 Saved Games/slot2/5051.ent - hash 4131653557338930076, mtime: 1384109141 11-11 21:34:27:534 csync_statedb_get_stat_by_hash: No result record found for phash = 4131653557338930076 11-11 21:34:27:534 _csync_detect_update: file: Basilisk Games/Book 2 Saved Games/slot2/5051.ent, instruction: INSTRUCTION_NEW <<=

. . .

11-11 21:45:40:225 _csync_merge_algorithm_visitor: INSTRUCTION_NEW file: Basilisk Games/Book 2 Saved Games/slot2/5051.ent

. . .

11-11 22:28:44:650 _csync_push_file: Remote repository atomar push enabled for owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2/5051.ent (0). 11-11 22:28:44:650 oc_module: => open called for owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2/5051.ent 11-11 22:28:44:650 oc_module: Stating directory owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2 11-11 22:28:44:650 oc_module: owncloud_stat owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2 called 11-11 22:28:44:734 oc_module: Simple propfind result code 207. 11-11 22:28:44:735 oc_module: => Errno after fetch resource list for owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2: 0 11-11 22:28:44:735 oc_module: Working on file slot2 11-11 22:28:44:735 oc_module: STAT result from propfind: slot2, mtime: 1384198653 11-11 22:28:44:735 oc_module: Directory of file to open exists. 11-11 22:28:44:735 oc_module: PUT request on /owncloud/remote.php/webdav/Basilisk%20Games/Book%202%20Saved%20Games/slot2/5051.ent! 11-11 22:28:44:735 oc_module: Sendfile handling request type PUT. fd 26 11-11 22:28:44:735 hbf_splitlist: hbf_splitlist: block_size: 10485760 threshold: 10485760 st_size: 121

11-11 22:28:44:735 hbf_splitlist: hbf_splitlist: num_blocks: 1 rmainder: 121 blk_size: 10485760

11-11 22:28:44:735 hbf_splitlist: hbf_splitlist: created block 0 (start: 0 size: 121) 11-11 22:28:44:735 oc_module: about to send 1 block 11-11 22:28:44:735 _hbf_dav_request: HBF: Block: 0 , Start: 0 and Size: 121 11-11 22:28:44:736 oc_module: Chunk upload completed for 'owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2/5051.ent' (121 bytes out of 121) 11-11 22:28:45:204 oc_module: Chunk upload completed for 'owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2/5051.ent' (372 bytes out of 121) 11-11 22:28:45:204 _csync_push_file: file: /Users/mario/owncloud-icepalace/Basilisk Games/Book 2 Saved Games/slot2/5051.ent, command: sendfile, error: Unknown error: 10014 from errno 10014 11-11 22:28:45:205 _csync_push_file: remember chunk: 0 (transfer id 348914414 ) 11-11 22:28:45:205 _csync_report_parent_error: Mark parent directoy Basilisk Games/Book 2 Saved Games/slot2 as an error 11-11 22:28:45:205 _csync_report_parent_error: Mark parent directoy Basilisk Games/Book 2 Saved Games as an error 11-11 22:28:45:205 _csync_report_parent_error: Mark parent directoy Basilisk Games as an error

. . . at about 3 a.m. there was a disconnect from the server! . .

11-12 03:23:48:889 csync_ftw: Uniq ID from Database: Basilisk Games/Book 2 Saved Games/slot2/5051.ent -> 11-12 03:23:48:889 csync_walker: file: /Users/mario/owncloud-icepalace/Basilisk Games/Book 2 Saved Games/slot2/5051.ent 11-12 03:23:48:889 _csync_detect_update: ==> file: Basilisk Games/Book 2 Saved Games/slot2/5051.ent - hash 4131653557338930076, mtime: 1384109141 11-12 03:23:48:889 csync_statedb_get_stat_by_hash: No result record found for phash = 4131653557338930076 11-12 03:23:48:889 _csync_detect_update: file: Basilisk Games/Book 2 Saved Games/slot2/5051.ent, instruction: INSTRUCTION_NEW <<=

. . .

11-12 03:43:54:047 _csync_merge_algorithm_visitor: INSTRUCTION_NEW file: Basilisk Games/Book 2 Saved Games/slot2/5051.ent

. . .

11-12 04:23:57:801 _csync_push_file: continuation: 0 348914414 11-12 04:23:57:803 _csync_push_file: Remote repository atomar push enabled for owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2/5051.ent (0). 11-12 04:23:57:803 oc_module: => open called for owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2/5051.ent 11-12 04:23:57:803 oc_module: Stating directory owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2 11-12 04:23:57:803 oc_module: owncloud_stat owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2 called 11-12 04:23:57:890 oc_module: Simple propfind result code 207. 11-12 04:23:57:890 oc_module: => Errno after fetch resource list for owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2: 0 11-12 04:23:57:890 oc_module: Working on file slot2 11-12 04:23:57:891 oc_module: STAT result from propfind: slot2, mtime: 1384198653 11-12 04:23:57:891 oc_module: Directory of file to open exists. 11-12 04:23:57:891 oc_module: PUT request on /owncloud/remote.php/webdav/Basilisk%20Games/Book%202%20Saved%20Games/slot2/5051.ent! 11-12 04:23:57:891 oc_module: Sendfile handling request type PUT. fd 26 11-12 04:23:57:891 hbf_splitlist: hbf_splitlist: block_size: 10485760 threshold: 10485760 st_size: 121

11-12 04:23:57:891 hbf_splitlist: hbf_splitlist: num_blocks: 1 rmainder: 121 blk_size: 10485760

11-12 04:23:57:891 hbf_splitlist: hbf_splitlist: created block 0 (start: 0 size: 121) 11-12 04:23:57:891 oc_module: about to send 1 block 11-12 04:23:57:891 oc_module: Existing chunk info 0 348914414 11-12 04:23:57:891 _hbf_dav_request: HBF: Block: 0 , Start: 0 and Size: 121 11-12 04:23:57:892 oc_module: Chunk upload completed for 'owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2/5051.ent' (121 bytes out of 121) 11-12 04:23:58:322 oc_module: Chunk upload completed for 'owncloud://icepalace.dyndns.org:81/owncloud/remote.php/webdav/Basilisk Games/Book 2 Saved Games/slot2/5051.ent' (372 bytes out of 121) 11-12 04:23:58:323 _csync_push_file: file: /Users/mario/owncloud-icepalace/Basilisk Games/Book 2 Saved Games/slot2/5051.ent, command: sendfile, error: Unknown error: 10014 from errno 10014 11-12 04:23:58:323 _csync_push_file: remember chunk: 0 (transfer id 348914414 ) 11-12 04:23:58:323 _csync_report_parent_error: Mark parent directoy Basilisk Games/Book 2 Saved Games/slot2 as an error 11-12 04:23:58:323 _csync_report_parent_error: Mark parent directoy Basilisk Games/Book 2 Saved Games as an error 11-12 04:23:58:323 _csync_report_parent_error: Mark parent directoy Basilisk Games as an error

. . .

This is the block for exactly one file.

EDIT:

mysql> select path, etag from oc_filecache where path like '%5051.ent%'; Empty set (0.50 sec)

It seems that the entry for this file never gets written to the DB...

Combustible commented 10 years ago

Owncloud client version: 1.4.1 (Linux Mint 14) Owncloud server version: 5.0.10 (Apache2 / PHP5, Ubuntu Server 13.10) Using only default owncloud apps, (not using encryption application)

I'm experiencing this problem too, though I'm hoping that I have a situation that is easier to debug.

I currently have five text files that are re-downloaded by the Windows owncloud client every 5 minutes. Fortunately, these are all text files, totaling just a few KB. Here's how I got to the point I'm at now:

First, I noticed that these files are re-downloading repeatedly. I poked around my oc_filecache table in the database, and noticed there are several old, stale files in there. So, I followed these steps:

Close client
Move files that are redownloading to a temporary location
Delete entire sync folder from client machine, so it will re-download (only 90 MB)
Stop server
Delete files manually from owncloud data store
Truncate owncloud oc_filecache table
Rebuild the owncloud oc_filecache table using the command "php console.php files:scan --all"
Examine oc_filecache, looks good. No references to these files.
Start server
Start client. Files re-download, as expected. These redownload files are not present, which is expected because they have been removed from server.
Copy the 5 files in question back to the sync folder. They get uploaded, as expected.

Now, the mystery -> These files are still redownloaded every 5 minutes. Looking at the owncloud client log, I see entries like: 11-12 15:21:27:626 _csync_detect_update: ==> file: SciTEUser.properties - hash 5614351735128935507, mtime: 1370734120 11-12 15:21:27:626 _csync_detect_update: Database entry found, compare: 1370734120 <-> 1370734120, md5: 528155b679a99 <-> 528155b679a99-gzip, inode: 0 <-> 255565 11-12 15:21:27:626 _csync_detect_update: file: SciTEUser.properties, instruction: INSTRUCTION_EVAL <<=

Here's the oc_filecache entry for this file: (fileid, storage, path, path_hash, parent, name, mimetype, mimepart, size, mtime, encrypted, etag, unencrypted_size) VALUES (39175, 12, 'files/CloudFiles/SciTEUser.properties', 'd2eaab5e737b70ea3d0be9cdb63b93aa', 17671, 'SciTEUser.properties', 14, 13, 3146, 1370734120, 0, '528155b679a99', 0);

This particular file is only 3146 bytes. Other files I have that have similar issues are of size 367 bytes, 3254 bytes, 298 bytes, 1481 bytes.

So, it looks like this is the same issue described earlier -> for some reason, the md5 of the server is being compared with the local md5 which contains a -gzip string. Looking in the oc_filecache table, the etag does NOT contain the -gzip string. In fact, no files in the server's oc_filecache table contain a -gzip in the etag.

Interestingly, it isn't possible to modify or delete these files now that they're added. I assume this is because upon modification, the database is comparing the journaled etag against the server and finding them different, assuming the server has an updated version, and then marking local changes as a conflict to that change.

On a completely different Windows system, I'm experiencing the same behavior. I tried updating to 1.4.2, deleting the sync folder and redownloading, but those same 5 files still download themselves over and over. It doesn't seem to be every 5 minutes, but I still can't delete or modify any of these 5 without triggering a redownload of all 5.

During this process, I have also added other files, which work fine (no repeated downloads)

So, as I'm sitting here writing this, these files are continuing to re-download themselves. To me, this doesn't look like a server issue, because the files in the cache all look like they're correct. Since this is a prime debugging opportunity, I'd like to lend what support I can.

Combustible commented 10 years ago

I have some more information to add after returning home (was at work).

So, I have these two computers: Linux home system running 1.4.1, on same network as server Windows work laptop, now 1.4.2.

I went home, and found that my home system is in fact behaving correctly, not re-downloading files. I can modify one of the five aforementioned files, and it is uploaded correctly. Now, I fired my work laptop up through the company VPN, and I'm still experiencing this issue. The one file I modified was downloaded (along with the other four...), and contains the change I made from home. However, I still can't modify any of these 5 files, without changes being converted into a conflict. I can however modify a different text file in the same folder, which causes it to upload and the 5 badly behaved files to redownload.

I'm kind of stumped. The well-behaved home system is synchronized to the same ntp pool as the server, but the work system likely has slightly different time (no idea how work does NTP). There's also proxy settings at work... Otherwise, same user, same files, different results. Anything useful I could try?

chrismyers81 commented 10 years ago

I know this theoretically shouldn't make a difference, but -

I have 4 pcs set up to sync against my oc server, two win7, one openSuSE 12.2, one oS 12.3. All were doing this, but with different files. I set the server to do a time sync every hour with a public time server. When a client would start doing the re-download over and over again, I would set it up against the same ntp server, and the re-syncs stopped (without rebooting, restarting the server/client, etc.) and haven't occurred again. They used to do the resync constantly but don't at all anymore, and it's been a couple of weeks for the first client I tried this on, a few days on the 3rd. (The 4th pc is rarely used, and not by me, so I don't pay much attention to it.)

Currently on OC 6 beta 2 on openSuSE 12.2, latest clients on win7, oS 12.2 and 12.3.

On Nov 12, 2013, at 10:38 PM, Byron Marohn notifications@github.com wrote:

I have some more information to add after returning home (was at work).

So, I have these two computers: Linux home system running 1.4.1, on same network as server Windows work laptop, now 1.4.2.

I went home, and found that my home system is in fact behaving correctly, not re-downloading files. I can modify one of the five aforementioned files, and it is uploaded correctly. Now, I fired my work laptop up through the company VPN, and I'm still experiencing this issue. The one file I modified was downloaded (along with the other four...), and contains the change I made from home. However, I still can't modify any of these 5 files, without changes being converted into a conflict. I can however modify a different text file in the same folder, which causes it to upload and the 5 badly behaved files to redownload.

I'm kind of stumped. The well-behaved home system is synchronized to the same ntp pool as the server, but the work system likely has slightly different time (no idea how work does NTP). There's also proxy settings at work... Otherwise, same user, same files, different results. Anything useful I could try?

— Reply to this email directly or view it on GitHubhttps://github.com/owncloud/mirall/issues/994#issuecomment-28363823 .

dragotin commented 10 years ago

@Combustible would you be able to use for example wireshark to record the traffic between the client and the server to see where the -gzip is added? That would be extremely helpful.

flurischt commented 10 years ago

I was creating a log for #986 when this problem started again. There are a lot of "11-12 23:06:00:440 _insert_metadata_visitor: SQL statement: INSERT INTO metadata_temp (phash, pathlen, path, inode, uid, gid, mode, modtime, type, md5) VALUES (658284358777750, 41, InstantUpload/IMG_20130615_225102 (2).jpg, 24246374, 501, 20, 33188, 1371922007, 0, 51c5de58b16050.60512244);" entries in it and the client now wants to resync ~10000 files with a total of 6 GB. I have a 18MB log but I don't want to post this online as there is my private server url and the names of my files in it. If a dev is interested in it, let me know how to contact you.

Combustible commented 10 years ago

@dragotin It took me a while, but I managed to get what will hopefully be a useful traffic record of this process.

For the sake of readability, I'm going to define "good" files as files that upload/download correctly, and "bad" files as those which re-download themselves and cannot be modified by the client without generating conflicts.

Since normally my traffic is all SSL protected, I had to switch that off to get meaningful a meaningful packet capture. In an attempt to reduce the amount of non-SSL data I sent out, I created a new garbage user, and shared a few "good" files with him, as well as several "bad" files. It took me several tries, but I think I managed to replicate this problem on this user without having to include a bunch of other noise.

Here's the process that eventually worked:

Stop client
Using web UI, create TESTUSER
Share some "good" and "bad" files with TESTUSER
Start client with log window, configure TESTUSER account.
All files sync, clear log window / wireshark packet capture.
From web UI, share an additional "good" file with TESTUSER
Client triggers download of that "good" file, as well as two "bad" files which should not have been downloaded.

I have the wireshark capture and the owncloud log of that last step. It's really interesting, because it looks like it's the server that is for some reason appending -gzip to the etags of these "bad" files.I have again verified that the oc_filecache table does not contain any entries with -gzip in the etag field. I am still unable to generate additional "bad" files, and I have no idea why these are unique. As I mentioned earlier, I can modify them fine from the client on my home machine.

I need an email address to send these captures to. They don't contain overly much sensitive info, but more than I'd like to post here.

dragotin commented 10 years ago

@Combustible would you mind sending the wireshark log to freitag at owncloud.com ? That would be helpful :)

DeepDiver1975 commented 10 years ago

@Combustible as far as my research goes mod_deflate is appending -gzip. Do you have mod_deflate enabled? Can you give it a try with disabled mod_deflate?

THX

etiess commented 10 years ago

Hello @DeepDiver1975

I have this issue too (redownloading): how can I check if mod_deflate is enabled?

If I use this command: apache2ctl -t -D DUMP_MODULES It returns a list of modules, and "deflate_module (shared)" is there

I don't see anything in apache.conf, and nothing in .htaccess (the one in the owncloud directory) In /etc/apache2/conf-enabled , there is a "deflate.conf" and a "deflate.load"

If I test my server ("http://mypersonalserver.com/owncloud") on http://www.whatsmyip.org/http-compression-test/ the result is "Your server is gzipped"

If this is the issue, I think that https://github.com/owncloud/core/issues/4783 should be really considered.

Thank you!

Bude132 commented 10 years ago

Hello @etiess

Well you basically answered your question yourself.

"apache2ctl -t -D DUMP_MODULES" if the module is listed there, it is enabled.

To disable this module use the following commands:

a2dismod deflate service apache2 restart

etiess commented 10 years ago

Thanks. I disabled it: does it have other consequences? @karlitschek commented in https://github.com/owncloud/core/issues/4783 :

Nowadays we no longer do compression with php so mod_deflate should be ok. On my test machine mod_deflate is activated and seems to work fine.

masterblaster79 commented 10 years ago

This simply disables compression and therefore increases the amount of data to be transferred (depending on the content) - so this can be considered as a workaround only as http compression is a good thing and best practice imho. Btw: Enabling mod_deflate is not enough to enable compression, somewhere in the configuration an output filter has to be set: "SetOutputFilter DEFLATE" Could be that the developer had mod_deflate enabled but nevertheless was using no comression so he didn't recognize the issue.

To disable compression, you can simply set "SetEnv no-gzip" and/or "RequestHeader unset Accept-Encoding" for your owncloud virtualhost or directory, no need to disable the module globally.

As soon as I have some time I'll check and confirm back if this mitigates the issue for me.

Cheers.

masterblaster79 commented 10 years ago

Good catch guys ;-)

Apache indeed adds the -gzip to the etag response header when compression is enabled. https://issues.apache.org/bugzilla/show_bug.cgi?id=39727

BrowserMatch "mirall" no-gzip BrowserMatch "csyncoC" no-gzip for the owncloud vhost solved the problem for me.

This can be fixed in the client by ignoring the trailing "-gzip" in the etag header.

Edit: Don't know if the Android-Client is affected as well, if yes also BrowserMatch "Android-ownCloud" no-gzip should be added.

Cheers, Stephan

fossxplorer commented 10 years ago

Has the issue boiled down to two different issues? Unnecessary downloads and ETAGs with -gzip? Because the initial issue does exist on Nginx too!

dragotin commented 10 years ago

@fossxplorer true, we only have an explanation for the -gzip issue yet. The other one with the changing etags is still open, and urgent.

Combustible commented 10 years ago

@masterblaster79 Thanks for the info!

It looks like all of my misbehaved files are being gzipped by the server, whereas well behaved files are not. I don't really understand the heuristic the server is using to determine which files are well suited for compression, but I guess I don't really care.

So, to summarize, it seems like my issues are entirely caused by the server choosing to gzip some files. When the client asks the server for changes, the server reports back etags, some of which don't match the client's etag because they are gzipped and contain -gzip appended to the etag. The client then re-downloads all of those files.

If your server aggressively zips things (it seems mine does not), this would result in a ton of extra traffic.

I can confirm that by adding the three BrowserMatch lines @masterblaster79 recommended to my virtual host and then restarting apache has solved this problem for me. I did not have to restart clients to see the fix take immediate effect at the next sync.

Awesome! It would be nice, however, if the client supported this compression mechanism a little better :)

fossxplorer commented 10 years ago

@dragotin Ok, thanks for the confirmation. I agree that this is in fact an urgent issue. FYI i'm using 1.3 with OC server 5.0.13 and the download issue is not present. This makes me believe the the issue must have been introduced after version 1.3 and it's limited to client-side only. But i could be wrong here!

jochenwezel commented 10 years ago

[...] ETAGs with -gzip [...] Awesome! It would be nice, however, if the client supported this compression mechanism a little better :)

yes, client support is required to handle this correctly (fixing with BrowserMatch Rules on server doesn't solve the problem for all cases, because there are often used some proxies or reverse proxies in the middle - doing some gzipping independently from owncloud server settings)

masterblaster79 commented 10 years ago

true, we only have an explanation for the -gzip issue yet. The other one with the changing etags is still open, and urgent.

Apache changes the etag by adding "-gzip" to it, other servers might behave different. Could be that for example nginx simply calculates a completely different etag when compression is used. "changing etags" could be the same issue still (Other webserver than Apache, or proxies adding compression as Jochen said).

I don't really understand the heuristic the server is using to determine which files are well suited for compression, but I guess I don't really care.

I added the DEFLATE output filter manually to compress everything. Could be that your distributions deflate default configuration only compresses some files based on the mime type. That might be an explanation why in your case only some files are affected.

ways commented 10 years ago

I upgraded to ownCloud 6 beta 3. Have not seen the issue since.

Edit: forget it, it's still there.

mkormendy commented 10 years ago

After seeing this issue and understanding what is involved,.. The best place to address a workaround would be in the mirall client.

gzip compression is important for reducing traffic between server and client (regardless of the client being used, browser or otherwise).

However due to the different server setups, apache patches, and settings of gzip turned off or on, it would be safer to say that we can't simply force everyone to turn off gzip for their ownCloud server. I don't have specific statistics behind the overall effectiveness of having gzip compression, but, however negligible it may be,... Wouldn't it just be overall easier and straightforward to add a condition that removes the suffixed "-gzip" from the etag comparison and then be done with this issue in the thread?

dragotin commented 10 years ago

@mkormendy we could add this condition maybe, but as said above, this problem is NOT the main one here. We're not at all done here.

mkormendy commented 10 years ago

@dragotin you're right we're not done here, by a long shot! But, for the large group of us affected only by the present gzip issue, can we get a new subsequent version publish of the latest code that has the conditional statement that works around the gzip issue in the meantime?

rhocco commented 10 years ago

Hi have the same Bug, that the client is redownloading random files from the server again. I have ~16gb of data and the server has an upload of 45kb/s (i know its very bad). When i am at the local LAN network i synced all the 16Gb with oc5.0.13 and 1.4.2 client in few minutes. Now over the internet nearly every day there have to be synced again ~200 mb ... after a week i have now to sync 12GB over the internet. Sadly now i cant use this. Hope you fix this bug in high priority.

chrismyers81 commented 10 years ago

@rhocco - try to sync all your clients and server to the same NTP server. It worked for me, even though it's not supposed to make a difference.

rhocco commented 10 years ago

@chrismyers81 ok, i had a difference in 43 seconds , now i set up same NTP on all windows clients (desktop & laptop) and my debian server. I try next time i am in my local LAN for faster sync. thank you for the tip i was not thinking about.

jaydcarlson commented 10 years ago

I haven't read through ALL the comments, but if this hasn't been said: for me, it seems like the re-downloading only happens when I login to the web interface. None of my users have been logged into the web site for a week, and haven't had any problems with this issue since.

BenS89 commented 10 years ago

I can confirm this. Without login to the web interface, client does not sync the whole root directory.

corazza commented 10 years ago

I can confirm this bug as well. For me, the client randomly starts re-downloading files that I already have, and have not changed from any system.

As far as I can see, this only happens to my Images/ directory. But since that's the largest one, it could be that I'm just not paying attention when it's downloading the others.

Does anyone know when can we expect a fix for this?

rhocco commented 10 years ago

@yannbane this issue is up for 3 months now. I think they focus on other bugs. I cant imagine the professional version with this bug in a company ... therefore i switched back to dropbox. i can confirm this behaviour too with login into the web interface. Again nearly the complete directory has to be re-downloaded again on my system (12GB) , sync time on Server/Clients dont helped. Remove client based compression for mirall in virtual host in apache dont helped too.

kwillems commented 10 years ago

Just compiled a new build. I can confirm https://github.com/owncloud/mirall/issues/1195 solved the unnecessary downloads in my case (client running on mac and server on linux and ownCloud 5.0.13).

dragotin commented 10 years ago

@kwillems thanks for letting us know.

mambo123 commented 10 years ago

Hi,

Great work on owncloud but I would like to give a feedback on this issue also.

Server OS: Debian GNU/Linux 2.6.32.11-svn70860 owncloud server: 5.0.13 Client OS: Windows 7 Enterprise 64-bit owncloud client: 1.4.2

Issue observation: Some new files on my client, so the files get uploaded when I powered on my owncloud server. The new files get uploaded successfully. Then I logged in to the owncloud web UI and clicked on the folder that was synced. It was at this moment that I noticed the owncloud client started to sync and download my entire files collection (9 GB).

I'm not sure if this is the root issue. Hope this helps you to investigate the issue.

nlurkin commented 10 years ago

Hello, It just happened to me again. I opened the main page of the OC UI (with automatic connection so I directly see all my folders) then I closed the page as soon as it was loaded without clicking anything else (I do it from time to time to trigger the eventual updates of OC). And at the next client sync, I notice that it's downloading 3.8GB of files again (and that's only in 1 of my synced folder, I need to wait for this one to finish before seeing if the second is also redownloading). I hope this will be solved with the new OC6, because 4GB of useless download several times a month is a huge waste of bandwidth, especially combined with the other issue I reported (#1154)

jochenwezel commented 10 years ago

I already realized at another issue, that as soon as you login at web UI, owncloud is doing some "magic" in background (e.g. recieved share permissions from other users sometimes appear on client devices only after opening the web UI first). Maybe there is a recalculation on tree that happens as soon as you open the web UI? (BTW: this "magic" should better run always so that client devices see the new shares from other users without opening the web UI - but this would be another ticket ;-)

ecloudbizsolns commented 10 years ago

Not sure if my issue is related or not, but the net effect is the same as above. There were several file permission, path length, and windows issues uncovered during the course of my investigation that may help others, so sorry for long post but there are a few things that all contributed in some way I believe (at least, fixing them all has resolved my issue...).

For the impatient:

ensure all file paths do not exceed max length on client (1), else file/folder re-creation will fail (despite syncing from there as source)
ensure the read-only flag is off (I'm not certain on this but it was across many of the problem files)
ensure the temporal flag is off - need utils (powershell or free download) to do this.
finally and most importantly, ensure windows does not block access to the file. You will need free utilities for this too.

(1) 250 odd characters on windows, my client had some over 275... through archiving folders repeatedly over the years

I've been trying to sync 30GB for the past week or so. The initial sync would complete successfully but afterwards it would randomly down/upload large chunks (>>1GB). Over several hours later this had reduced to a seemingly small (between say 10-20) list of files that simply would not complete the sync. The error was "Error in directory". I checked and rechecked the server side suspecting it to be the issue, as the data on the client had previously been synced with another tier 1 provider for 3+ years and so I did not suspect the client side.

Build is 5.0.12 community on FreeNAS 9.1.1-Release (installed into a BSD jail). All inside a VM on ESXi5, HP G6, 24GB ram, HP hardware RAID10.

Edit: Client is 1.4.2 on win 7

This morning I had all but given up on using oc for anything serious as the sync was proving to be very unreliable, repeatedly up/downloading the same set of files.

I had found several paths that exceeded the windows limit, and repaired all these.

The issue persisted.

I took a look at the file permissions on the client - the files in question had NTFS permissions RATI - wtf is T? A search revealed numerous other read-only files that were not reported, so I assumed the Read Only flag is irrelevant, but I'm not certain on that point (someone will advise...). I removed the readonly flag to be sure.

Removing the T flag proved non-trivial. On Win7 Pro the attrib command is unable to manipulate it... wind blows.

Digging found a post with a 2 line powershell remedy. http://community.spiceworks.com/scripts/show/1102-remove-temp-file-attribute

The issue still persisted.

It should be noted here that no other systems have access to the server at this point, and no files are being changed on the client. The server timezone and client are the same and they are synced with ntp (and the time isn't drifting too quickly, even though a VM).

Examing the properties of the few remaining files yielded the final fix - the files were being blocked by windows! with the "This file came from another computer..." ...excuse. At least, unblocking it resulted in that file syncing correctly the next time and not resync ever again unnecessarily.

It seems the files in question (all had this same flag) had been downloaded from a multitude of sources, dropbox, yousendit, email, web sites, and windows had tagged them as special. Yeah, thanks...

The fix involves removing the alternate data streams from all files in client repo (which contain the location information info that allows windows to silently block access to owncloud sync client while simultaneously allowing you full access to it - go figure that out).

And finally, after many many many hours over several days and nights, the sync is complete, there are no errors, and it does not and has not up/downloaded anything without undue reason. I am dancing on the ceiling now :) And I have opened up access to the business who is trying to recover this info and their sync is complete and is also no longer up/downloading unnecessarily.

For my case, removing these alternate streams was also non-trivial, "streams.exe -s -d *" (from SysInternals Suite) failed with permissions errors (imported cloned customer failed hdd...) Luckily I found: http://www.nirsoft.net/utils/alternate_data_streams.html ...which provided the virtually instant fix to that (my account already had Full Permission on the whole Documents subfolder, member of admin group, etc. Wind blows remember).

Hopefully this info may help some others with these sync issues.

PS: The T attribute if present (and/or relevant) can also be removed using a tool from canon on XP only. Dies on Win7. Seems their photo sw used it somehow. Lost the link tho, sorry.

hdering commented 10 years ago

I can confirm the problem.

Server: 5.0.13 (Ubuntu 12.04) Client: 1.5.0beta2 (Windows 7)

karlitschek commented 10 years ago

Can you help us with a test please? Could you copy this installation ad uügrade to ownCloud 6 RC3 and check if this issue still exists? Would be great. Thanks

DeepDiver1975 commented 10 years ago

@karlitschek I just observed this issue again with stable6 - debugging into this once more .....

DeepDiver1975 commented 10 years ago

Steps for me to reproduce: 0.) copy an existing user folder within data (e.g. cp -R user user1) 1.) create a new user (e.g. user1) and assign a quota of 1GB 2.) setup owncloud client and connect user1 3.) wait until sync is finished 4.) share a folder with files with that user 5.) wait until sync is finished 6.) have a close look at the client's activity tab (on 1.5) 7.) first time login of that user via the web interface 8.) watch the client re-downloading all NON-shared files and folders

@karlitschek @dragotin @danimo @icewind1991

DeepDiver1975 commented 10 years ago

steps updated: the user has a 1GB quota - not sure if this is necessary - but I had this setup that way

DeepDiver1975 commented 10 years ago

UPDATE: it's not the login itself but the first ajax scan.php call - here is the sql trace: http://paste.debian.net/69491/

summoning support forces @PVince81 @ringmaster @bantu @bartv2

PVince81 commented 10 years ago

Can you quickly try without the quota ? It might have an influence. As long as the first scan hasn't been run, the "total size" isn't known. Not sure if it would affect the sync client.

PVince81 commented 10 years ago

Another thing to try is logging in using the web UI (and keep the sync client closed), then wait for scanFiles() to finish. Then start the sync client.

owncloud / client

client 1.4 unnecessarily downloads again those files which are already downloaded #994