search

owncloud / client

🖥️ Desktop Syncing Client for ownCloud
GNU General Public License v2.0
1.39k stars 668 forks source link

client 1.4 unnecessarily downloads again those files which are already downloaded #994

Closed jochenwezel closed 10 years ago

jochenwezel commented 11 years ago

A user shared with me a folder with about 10 10 GB data (approx. 200 files).

I don't know why, but it's often (not always) finding some new files. Download follows - but unnecessarily: files are absolutely identical (byte comparison); the file on server hasn't been touched for months. So the file hasn't changed.

Following suggestions:

  1. Finding and fixing the reason why the client trys to re-download equal files which hasn't been changed/updated in any way
  2. Before downloading files: download a hash of the file from server for that file + check if hash can be found in local journal database + if yes: copy the file locally

Local client: 1.4.0 on Win2008 R2 server (and other win systems, too) Server version: 4.5.7 on Linux

DeepDiver1975 commented 10 years ago

Another thing to try is logging in using the web UI (and keep the sync client closed), then wait for scanFiles() to finish. Then start the sync client.

I did that already - has no impact because the etag is changed during the scan operation.

ecloudbizsolns commented 10 years ago

My user had no quota asssigned, 40GB+ free disk. I definitely was logged into the web UI during the initial sync, checking its progress. Only two users, cust1 and admin. No shares. All files transferred under cust1 account.

@karlitschek was that question re upg to 6 to me? I can, might take a few days tho.

icewind1991 commented 10 years ago

Steps for me to reproduce: 0.) copy an existing user folder within data (e.g. cp -R user user1) 1.) create a new user (e.g. user1) and assign a quota of 1GB 2.) setup owncloud client and connect user1 3.) wait until sync is finished 4.) share a folder with files with that user 5.) wait until sync is finished 6.) have a close look at the client's activity tab (on 1.5) 7.) first time login of that user via the web interface 8.) watch the client re-downloading all NON-shared files and folders

I don't think the sharing is needed to trigger the error, the first time scan.php is called for a user seems to always re-generate etags

DeepDiver1975 commented 10 years ago

I think I've got it!

@icewind1991 Scanner::backgroundScan() is the root of evil - it is calling scan() without \OC\Files\Cache\Scanner::REUSE_ETAG - as a result existing etags in the cache are getting lost

dragotin commented 10 years ago

Hero points for @DeepDiver1975

icewind1991 commented 10 years ago

@DeepDiver1975 you beat me by a minute :)

https://github.com/owncloud/core/pull/6201

ways commented 10 years ago

@DeepDiver1975 Sounds promising!

mdieterich commented 10 years ago

Crossing my fingers, toes, and whatever else I can that this is the solution! It's been debilitating my users.

@DeepDiver1975 https://github.com/DeepDiver1975 Sounds promising!

blackm commented 10 years ago

DeepDiver1975 for president!!!

dragotin commented 10 years ago

@blackm well, don't exaggerate - better keep him fixing bugs than becoming a tie-guy ;-D

hdering commented 10 years ago

I would say beta3 :) or is it possible to get a special version to test it?

protree commented 10 years ago

Take a look at the patch posted by icewind1991, worked for me on owncloud 5.0.13. The file to change is "/owncloud/lib/files/cache/scanner.php" (The code-passage is at the end of the file)

dragotin commented 10 years ago

@hdering note that you need to fix the server to enjoy the fix. Stay tuned for the next oC6 RC or the next stable5 release, or pick the patch.

hdering commented 10 years ago

ohh okay...I thought it relates to the client. I test the patch now.

mdieterich commented 10 years ago

I've patched our OC 5.0.13 installation and will let people know if I continue to see the issue.

Thanks!

Mark

@hdering https://github.com/hdering note that you need to fix the server to enjoy the fix. Stay tuned for the next oC6 RC or the next stable5 release, or pick the patch.

— Reply to this email directly or view it on GitHub https://github.com/owncloud/mirall/issues/994#issuecomment-29927675.

fossxplorer commented 10 years ago

@DeepDiver1975 you the man!

mkormendy commented 10 years ago

Great work @DeepDiver1975 . I don't presume this fixes the gzip issue as well ... so @dragotin is there a patch on the client that takes care of the separate gzip issue?

DeepDiver1975 commented 10 years ago

Thx.  Afaik the gzip issue is fixed for 1.5

Von Samsung Mobile gesendet

dragotin commented 10 years ago

On 05.12.2013 22:37, Thomas MĂĽller wrote:

Thx. Afaik the gzip issue is fixed for 1.5

Yes, the gzip issue does not happen with 1.5.0 beta any more.

mkormendy commented 10 years ago

Pardon me for asking a likely obvious question,.. but where do I d/l the new beta?

dragotin commented 10 years ago

@mkormendy come on, how long are you around with us? ;-)

It's http://owncloud.org/sync-clients/#testing

mkormendy commented 10 years ago

Beat me to the punch .. just found it. Really I never knew, thanks either way drag!

dragotin commented 10 years ago

So, I take the pride to close this (probably) mostly discussed ownCloud bug. Kudos to @DeepDiver1975 and @icewind1991

csware commented 10 years ago

I still see this issue with Mirall 1.5 on Windows 7 x64.

jaydcarlson commented 10 years ago

@csware did you apply the patch to your server? It's not an issue with the client; it's an issue with the server. I can verify that the proposed patch fixes the problem -- I've had no further issues with it since I applied the patch to my installation.

csware commented 10 years ago

@funkathustra Which patch?

jaydcarlson commented 10 years ago

@csware The one mentioned in this thread? Here's a link: https://github.com/owncloud/core/pull/6201

This issue has been fixed and closed. If you're still having issues after applying this patch, you should create a new issue.

danimo commented 10 years ago

@csware the patch is included in 5.0.14(a) and 6.0.0(a)

chrismyers81 commented 10 years ago

If I'm currently on 6.0.0rc4, is it ok to go to 6.0.0a? Wasn't sure if that would be a downgrade or not?

.:. Sent from my phone, please forgive my typing .:.

On Dec 21, 2013, at 2:07 AM, Daniel Molkentin notifications@github.com wrote:

@csware https://github.com/csware the patch is included in 5.0.14(a) and 6.0.0(a)

— Reply to this email directly or view it on GitHubhttps://github.com/owncloud/mirall/issues/994#issuecomment-31059039 .

chrismyers81 commented 10 years ago

Nevermind, I just answered my own question. When I went to install 6.0.0a, it said that the installed 6.0.0rc4 was newer, and wouldn't let me install it. I didn't want to do the --force option since I didn't know what that would do to the database if I installed an older version. So I guess I'll wait for rc5 to come out and hope that that includes the fix.

jaydcarlson commented 10 years ago

@chrismyers81 the patch is fairly trivial -- it only requires changing one line of code in a specific file (details mentioned previously). You could easily patch your live codebase without reinstalling owncloud. That's what I did, at least.

chrismyers81 commented 10 years ago

Hmm...

Is the patch this one: https://github.com/owncloud/core/pull/6201 ?

If so, it looks like 6.0.0RC4 has it already, but my win7 machine here at home keeps pulling the same 6 files over and over again still :/

chrismyers81 commented 10 years ago

The weird thing about the files it keeps trying to sync - on the logger screen on the client (1.5.0), it says they're all 0 bytes. However, on the server, they're not, and when they were downloaded originally, they are the correct size. The Apache logs have them listed as the correct size as well.

[21/Dec/2013:16:15:46 -0600] 192.168.1.103 TLSv1.2 DHE-RSA-AES256-GCM-SHA384 "GET /owncloud/remote.php/webdav/Music/JJ_Heller/The_Pretty__The_Plain/Tonight.mp3 HTTP/1.1" 5063626 "-" "Mozilla/5.0 (Windows) csyncoC/0.91.4 neon/0.30.0"
[21/Dec/2013:16:15:47 -0600] 192.168.1.103 TLSv1.2 DHE-RSA-AES256-GCM-SHA384 "GET /owncloud/remote.php/webdav/Music/JJ_Heller/The_Pretty__The_Plain/When_You_Come_Back.mp3 HTTP/1.1" 2202915 "-" "Mozilla/5.0 (Windows) csyncoC/0.91.4 neon/0.30.0"
[21/Dec/2013:16:15:47 -0600] 192.168.1.104 TLSv1.2 DHE-RSA-AES256-GCM-SHA384 "PROPFIND /owncloud/remote.php/webdav/Work HTTP/1.1" 380 "-" "Mozilla/5.0 (Linux) mirall/1.5.0"
[21/Dec/2013:16:15:47 -0600] 192.168.1.103 TLSv1.2 DHE-RSA-AES256-GCM-SHA384 "GET /owncloud/remote.php/webdav/Music/JJ_Heller/The_Pretty__The_Plain/When_You_Come_Back_reprise.mp3 HTTP/1.1" 4672438 "-" "Mozilla/5.0 (Windows) csyncoC/0.91.4 neon/0.30.0"
[21/Dec/2013:16:15:48 -0600] 192.168.1.103 TLSv1.2 DHE-RSA-AES256-GCM-SHA384 "GET /owncloud/remote.php/webdav/Music/JJ_Heller/The_Pretty__The_Plain/Where_I_Land.mp3 HTTP/1.1" 5368973 "-" "Mozilla/5.0 (Windows) csyncoC/0.91.4 neon/0.30.0"
[21/Dec/2013:16:15:48 -0600] 192.168.1.103 TLSv1.2 DHE-RSA-AES256-GCM-SHA384 "GET /owncloud/remote.php/webdav/Music/JJ_Heller/The_Pretty__The_Plain/Why_Is_It_Colder.mp3 HTTP/1.1" 5185888 "-" "Mozilla/5.0 (Windows) csyncoC/0.91.4 neon/0.30.0"
[21/Dec/2013:16:15:49 -0600] 192.168.1.103 TLSv1.2 DHE-RSA-AES256-GCM-SHA384 "GET /owncloud/remote.php/webdav/Music/JJ_Heller/The_Pretty__The_Plain/You_Tell_Me_So.mp3 HTTP/1.1" 4777123 "-" "Mozilla/5.0 (Windows) csyncoC/0.91.4 neon/0.30.0"

The owncloud logs (on 'everything' level,) don't report anything.

chrismyers81 commented 10 years ago

I did a force-upgrade to 6.0.0a (which is newer than rc4 after all, I guess that the rpm command just interpreted the versions differently.) The win7 client in question still keeps trying to re-download those mp3 files.

jaydcarlson commented 10 years ago

Obviously, check your copy of lib/private/files/cache/scanner.php to make sure it has the changes mentioned in the patch. If you're sure your installation has the patch, and you're still having problems, open another issue. This issue has been fixed and closed, so this is not an appropriate thread to be discussing another issue (even if it has similar symptoms)

dschmidthawk commented 10 years ago

I'm having the same issue with one file, continually downloaded at 0kb. I've checked to make sure the patch changes are in place. I'm using Mirall 1.5 on Windows 7 and the latest server (6.0a). The download states 0kb but the file on the server 2.3kb.

10:10:17 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
10:02:47 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
9:56:36 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
9:50:35 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
9:44:46 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
9:36:28 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
9:29:48 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
9:23:19 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
9:16:39 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
9:09:55 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
9:03:21 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
8:56:38 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
8:50:05 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
8:43:31 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
8:36:53 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
8:30:19 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
8:23:46 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
8:17:18 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
8:10:42 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
8:04:12 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
7:57:42 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
7:51:14 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
7:44:38 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
7:38:01 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
7:31:24 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
7:24:44 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
7:18:19 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
7:11:39 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
7:05:09 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
6:58:42 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
6:52:02 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
6:45:30 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
6:38:48 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
6:32:19 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
6:25:47 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
6:19:20 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
6:12:41 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
6:06:13 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
5:59:40 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
5:53:13 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
5:46:50 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
5:40:13 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
5:33:47 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
5:27:18 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
5:20:43 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
5:14:13 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
5:07:41 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
5:01:08 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
4:54:39 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B
4:48:16 AM10_000 Maniacs/Love Among the Ruins/AlbumArtSmall.jpg Music Download 0 B

dschmidthawk commented 10 years ago

I'm still having the same problem with 1.51rc1 on Windows 7 with 6.01 stable server. Any word on this?
As a note, my lib/private/files/cache/scanner.php file contains the fix referred to above with etag.

2-9-2014 7-25-54 pm- 0bytefiles

fragtion commented 10 years ago

Same problem here

PVince81 commented 10 years ago

Are you guys using external storage ? Is it a Linux server ?

Not sure why the client is showing "of 0B", any idea @dragotin ?

dragotin commented 10 years ago

Maybe there is a hanging NFS server in the background? I

fragtion commented 10 years ago

I was just syncing a user's "Desktop" folder between two laptops running Win7 x64 (latest public v1.5.0 client, and v6.0.1 server from official repo), server is Ubuntu 13.10 i386 with latest stable packages installed. I wasn't seeing the 0B issue in the logs, files appeared to have normal sizes reported

In my case, I symlinked owncloud's htdocs's 'data' node to point to a folder store on a 3TB ext4 RAID-0 array, although I doubt that makes any difference. Permissions are correct (www-data:www-data) and everything else seems to work as normal

Once the initial sync completed between the two clients, some of the files (on the Desktop) and one of the folders, would repeatedly sync back and forth between them (uploading then downloading, causing a seemingly endless overwrite loop).

I tried deleting the user, removing the data files from the server, clearing the one client, and repeating the process, and then the same files went into a loop once the initial sync completed

Using btsync as a workaround so long and it has had no problems with the same sync task. Hoping to switch back asap as I prefer the client-server-client to the p2p topology among the many other awesome unique features that ownCloud provides

dragotin commented 10 years ago

@fragtion can you provide a log file snippet about the files in question, both client- and server access log?

dschmidthawk commented 10 years ago

I'm using a linux server through BlueHost. no external anything.

fragtion commented 10 years ago

Ok I've saved a log of it happening from one of the clients. It's 2.5MB uncompressed, so I couldn't pastebin it, rather a direct link to my server - http://197.242.147.241/owncloud-logs.zip . The server instance's log is really tiny - I hope I'm searching the right file. Does the server need to be running in a debug mode as well?

fragtion commented 10 years ago

Acknowledgement? I suppose this hasn't been identified and resolved yet in v1.5.1 client? Is it more likely a client issue?

dschmidthawk commented 10 years ago

I can attest to the fact that it isn't corrected for me in the latest version: 1.5.1 (build 2337)

inducer commented 9 years ago

Not fixed for me either.

obma commented 9 years ago

Reoccurred for me too.

Server: 7.0.4 (stable) - Ubuntu package Client: 1.7.1- Arch Linux AUR

seiferflo commented 9 years ago

Hi, Same problem for me. Random files being downloaded to the client for no reason as the file is already there in both places and hasn't changed. The new file on the client has a new "date created" which is problematic for me as I use this metadata a lot. Sometime we're talking about thousands of files. If I close the client, open it back, same, it goes on and on. If I now close again and remove the .csync_journal.db, then I get only a dozens of files downloaded and these files are named with an *_conflict After that, it works great. Basically I'm just deleting each db everytime I close owncloud to avoid having this issue again. Weird but couldn't find another workaround.

I don't really understand much as I'm not a skilled user, but if you need anything from me, just let me know.

Server: 7.3.5 (Synommunity package) Client: 1.7.1 b4382 windows 8.1

inducer commented 9 years ago

FWIW, I rebuilt the cache and resynchronized the server clock, and the issue hasn't reoccurred since.