search

owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.34k stars 2.06k forks source link

server did not acknowledge the last chunk. (No e-tag were present) #12725

Closed toadhall closed 8 years ago

toadhall commented 9 years ago

Environment: root server Server: Debian Wheezy Database: SQLite Client: Debian Testing OC-Version: 7.0.3 PHP-Version: 5.4.35-0

Since updating to 7.0.3 I keep on getting this error:

The item is not synced because of previous errors: The server did not acknowledge the last chunk. (No e-tag were present)

Worked fine in version 7.0.2.1 and only started appearing on random files (all of the small, between 40 and 70kb) in version 7.0.3.

Please let me know if there is anything else you need from me (specific logs, etc.)

Many thanks in advance!

PVince81 commented 9 years ago

Aha! "Database is locked": please try the following patches:

If anyone in this thread is also using SQLite, please also try with these patches. Thanks.

ghost commented 9 years ago

@sleepysmurf See this comment: https://github.com/owncloud/core/issues/12725#issuecomment-73228922

Edit Or try the patches from @PVince81 above :-)

tflidd commented 9 years ago

Is there anything I can do with a mysql-database? There are no further errors in owncloud.log, only the clients complains about the last chunk not being acknowledged. The two situations I experienced this error, I was uploading a new folder via the client (once a few files, maybe 10 in Win7, the other one a bigger folder about 400 files and 2.2 GB in size in OS X).

PVince81 commented 9 years ago

@tflidd it your case it might be a different issue. Did you try setting "loglevel" to 0 in config.php ? With a bit of luck it will show more output in "owncloud.log" at the time the error happens.

tflidd commented 9 years ago

@PVince81 : It is already in the debug mode. Just for clarity, I put everything into the template:

Steps to reproduce

Upload a new folder with some files in it via the oc-sync-client (Win or OS X)

Expected behaviour

Sync all files.

Actual behaviour

Syncs most of the files but at some it comes an error: Server did not acknowledge the last junk. Now, the clients searches the folders regularly repeating this error over and over again. Restart of client or apache didn't help. I didn't move the file so far. I used the web-interface to upload the file to a different location (outside the sync-folder). This didn't cause any problems.

Server configuration

Operating system: Ubuntu 14.04

Web server: Apache

Database: Mysql 5.5.41

PHP version: 5.5.9

ownCloud version: 7.0.4

Updated from an older ownCloud or fresh install: Updated

List of activated apps: - activity

The content of config/config.php:

<?php
define('DEBUG',true);
$CONFIG = array (
  'installed' => true,
  'dbtype' => 'mysql',
  'dbname' => 'xxxx',
  'dbuser' => 'xxxx',
  'dbpassword' => 'xxxxx',
  'dbhost' => 'localhost',
  'dbtableprefix' => '',
  'forcessl' => true,
  'blacklisted_files' => 
  array (
    0 => '.htaccess',
  ),
  'trashbin_retention_obligation' => 30,
  'allow_user_to_change_display_name' => true,
  'updatechecker' => true,
  'has_internet_connection' => true,
  'enablebackup' => false,
  'datadirectory' => '/var/www/owncloud/data',
  'version' => '7.0.4.2',
  'installedat' => '1325006970.14',
  'lastupdatedat' => '1329928732.51',
  'loglevel' => '0',
  'logdateformat' => 'F d, Y H:i:s',
  'log_rotate_size' => 504857600,

  'instanceid' => '507effec80781',
  'maintenance' => false,
  'theme' => '',
  'trusted_domains' => 
  array (
    0 => 'xxxxx',
    1 => 'xxxxx',
  ),
  'appcodechecker' => false,
  'appstoreenabled' => false,
  'asset-pipeline.enabled' => false,
  'overwritewebroot' => '/owncloud',
  'mail_from_address' => 'owncloud',
  'mail_smtpmode' => 'php',
  'mail_domain' => 'xxxx',
  'secret' => 'xxxx',
  'asset-pipeline.enabled' => true,
);

Are you using external storage, if yes which one: No

Are you using encryption: yes

Client configuration

Operating system: I have seen this with the OC Client, latest version 1.7.0 in Windows and OS X. Actually it's the OS X, client hanging.

Client Log

https://github.com/owncloud/core/issues/12725#issuecomment-73027794

ownCloud log (data/owncloud.log)

nothing!

asinteg-daehn commented 9 years ago

Hi,

have the exactly same problem with small files (< 2MB) - and won't spam up the thread with a full report, just:

Is it right that even OC 8 won't bring any benefit / solution for this major problem?

asinteg-daehn commented 9 years ago

Update: Currently only found a WORKAROUND: Renaming of all affected files.

This is very annoying, but a simple workaround.

Seems, that the database already contains messed up entries for the files - independent from MySQL and SQLite.

tflidd commented 9 years ago

@asinteg-daehn Thank you for the hint with the database. I checked my database and in my filecache table, I found that the keyfile is already indexed but not the file itself. The filename is 12_42_16.843.mat. I checked the folder for the keyfiles and there is indeed a keyfile. So I switched to the directory, and big surprise: There are more than 1000 files like these two (picked randomly, the sync-client is running in the background all the time since yesterday):

-rw-r--r-- 1 www-data www-data  8192 Feb  5 14:05 12_42_16.843.mat.ocTransferId999399742.part
-rw-r--r-- 1 www-data www-data  8192 Feb  5 10:57 12_42_16.843.mat.ocTransferId999969193.part

I just compared two of these files and they are identical. They seem to be encrypted (they start with: HBEGIN:cipher:AES-256-CFB:HEND------) The size of the original file:

8347880 24 -rwxrwxrwx   1 user  staff  8238 Jan 26  2012 12_42_16.843.mat

So the encrypted data can't contain the file. Would make sense with the problem of the last junk.

ghost commented 9 years ago

@asinteg-daehn

Is it right that even OC 8 won't bring any benefit / solution for this major problem?

See the comment here about this error: https://github.com/owncloud/core/issues/12725#issuecomment-73237082 and note that this is very likely not one "major problem" but rather many different problems resulting in the same error message.

ghost commented 9 years ago

Is anyone at the owncloud company working on this? Because I am getting this error since I updated to the latest version on January and in my opinion it is a serious bug, since one of the core features of owncloud is to provide a secure and reliable file synchronization. Well, and unfortunately it is not reliable right now, as I (as well as other users, see above) experience from time to time files that are just not synchronized from one computer to another. And in my opinion, this is a major issue, maybe even the one with highest priority.

PVince81 commented 9 years ago

This error message is too generic. Need more detail. Please post logs. Thanks.

sleepysmurf commented 9 years ago

The solution seems to be to update the database to the non light version On Mar 9, 2015 5:23 PM, "Vincent Petry" notifications@github.com wrote:

This error message is too generic. Need more detail. Please post logs. Thanks.

— Reply to this email directly or view it on GitHub https://github.com/owncloud/core/issues/12725#issuecomment-77945903.

Avalanchediver commented 9 years ago

This issue is making owncloud nearly unusable. It simply does not synchronize "all" my files. My only solution has been to eliminate folders from sync that have the files that will not sync. One drive has no issue syncing all my files...

themagicm commented 9 years ago

@PVince81 Logs have been attached, so if you want more, you need to be specific. Having something not sync or just resync over and over and over when the file is clearly there is a bug.

PVince81 commented 9 years ago

I meant everyone should post logs instead of complaining, because it is likely to be a different issue for everyone. For one it might be a missing etag in the database (did you actually check there?), for another one a failed final rename, for a third a wrong permission, etc, which could all cause the same error message to be displayed but be completely different bugs.

This is why I'd like the message to be changed to "Server error, please check server logs" instead of this misleading message making everyone believe they have the same issue.

If multiple people post their server logs and they do indeed look similar, then we have more data points to analyze the issue. Because so far this could not be reproduced locally (with OC 7.0.4+ and sync client 1.8.0). So the more information, the better.

If the logs are different then they need to be raised as separate tickets to be looked into separately, focusing on one issue at a time.

I hope you now understand why "I have the same issue" isn't useful, because it doesn't tell which of the X possible issues it could be.

Thank you.

PVince81 commented 9 years ago

I quickly scrolled through (currently on mobile) and it seems that some can already split into

These are already candidates to be moved to separate tickets later.

ghost commented 9 years ago

Maybe could be also added to the list:

See https://github.com/owncloud/documentation/issues/344#issuecomment-77316150 for some infos

After doing some tests it also seems that mod_deflate is changing the e-tag: https://github.com/owncloud/core/issues/4783#issuecomment-77319167. Could this also be related to some of the reports here?

PVince81 commented 9 years ago
themagicm commented 9 years ago

@PVince81 ok...got it! Makes sense.

The issue I have is, the file does get sync'd but for some reason, I still get the same error like the one in this thread. The file gets sync'd over and over and over... I'm running SQLite and OwnCloud 7.0.4.2 under CentOS Linux 6.6

I did try your patches #13832, #13831.

If you need more info, just ask, I'll provide. Thank you for your time!

Also, I'm not using encryption, disabled it because of some other error it was causing..dont remember which.

owncloud log shows: {"reqId":"54fc573b38ceb","app":"webdav","message":"Sabre\DAV\Exception\NotAuthenticated: No basic authentication headers were found","level":0,"time":"2015-03-08T14:05:47+00:00","method":"PROPFIND","url":"\/remote.php\/webdav"}

PVince81 commented 9 years ago

If you use SQLite it is likely that you will either get "database is locked" or "duplicate entry" when running parallel sync. I'd recommend to switch to MySQL, MariaDB or PostgreSQL if possible (there's a convert tool in OC 7: https://doc.owncloud.org/server/7.0/admin_manual/maintenance/convert_db.html)

If you don't, there are some potential fixes on the way to 8.0.3 (hopefully) that should significantly reduce the occurence of such race conditions.

themagicm commented 9 years ago

@PVince81

well.. I went cold turkey and migrated to MySQL. Then I moved everything from my Owncloud folder out so it can delete all files on the "cloud", then I'll put it all back. I'll see if moving to MySQL fixed it.

Also..dunno if its just me..but it seems faster accessing owncloud thru the web interface.

EDIT: Ok, I moved all files out of the owncloud dir, let it sync (delete the missing files)..then put them back in the owncloud folder so they could sync. same problem. It was with 1 specific file, so I renamed it and it transferred. These files are Quickbooks backup files, nothing special. I'll let this process run and sync for a few days to see if the problem is solved.

themagicm commented 9 years ago

Ok..upgrading to MySQL so far has resolved it. I do not get the problem where a certain file keeps getting sent over and over even though its been sent already. Hopefully this fixed it. I'll report back after a week of file syncs.

ghost commented 9 years ago

upgrading to MySQL is not a fix, that is just a workround, @themagicm . the guys at owncloud still have to solve this problem.

PVince81 commented 9 years ago

For SQLite "Database is locked" issues, go to: https://github.com/owncloud/core/issues/10566

rr-paras-patel commented 9 years ago

I am facing same issue. I think this issue is related to e-tag calculation. becoz 1) if i copy file with similar name like newfile.txt --> newfile_1.txt it don't work 2) if i rename file newfile_1.txt --> somethingnew.txt it works and don't give e-tag error.

might be this clue will helpful for developer. Thank you

PVince81 commented 9 years ago

Are you guys still seeing issues with 7.0.7 ? Many encryption issues were fixed since.

PVince81 commented 9 years ago

@PatelParas the issue you described sounds like https://github.com/owncloud/core/issues/11127. This was fixed a long time ago.

PVince81 commented 8 years ago

Anyone still having issues with 8.1.3 ?

"Database is locked" issues should be gone

themagicm commented 8 years ago

I am not experiencing this anymore.

Thank you for the direct email asking, I appreciate it!

-Alex

On Thu, 10/8/15, Vincent Petry notifications@github.com wrote:

Subject: Re: [core] server did not acknowledge the last chunk. (No e-tag were present) (#12725) To: "owncloud/core" core@noreply.github.com Cc: "themagicm" bartonekdragracing@yahoo.com Date: Thursday, October 8, 2015, 7:58 AM

Anyone still having issues with 8.1.3 ?

"Database is locked" issues should be gone

— Reply to this email directly or view it on GitHub.

ghost commented 8 years ago

@PVince81 Thats should be solved by the OC-ETag header introduced by oC 8.1.x

jaqfneto commented 8 years ago

I installed OC version 8.2.1 and i am unable to upload any file i put in my local folder.

I allways get the error "The server did not acknowledge the last chunk. (No e-tag were present)".

Via browser i have no problems to upload, and all uploaded files that way are automatically downloaded to the local folder in my desktop.

If i create a folder in my ownCloud local folder, the folder is also created in the server.

I am using OC Server v. 8.2.1 and OC client version 2.0.2 for windows

Thanks

dsalvanes commented 8 years ago

I am a Linux server Administrator and I use WHM & cPanel. In one account I have OwnCloud and I have this problem (The server did not acknowledge the last chunk).

I can solve it. It is a problem with ModSecurity. I have activate in ModSecurity Vendors, OWASP ModSecurity Core Rule Set (provider is cPanel). I desactivate it and problem is solved.

I hope this can help to someone.

edgar4s commented 8 years ago

2 years later:

image

Any concrete solution, anyone?

PVince81 commented 8 years ago

@310TN server logs please + OC version. This message can mean anything.

By the way, is anyone else still seeing such errors in v9.0.1 ?

giuppi commented 8 years ago

I'm getting this error. Clean install v9.0.1

can create folders from both pc and android app can upload files from web interface

this is the log when i try to upload something from the android app, nothing gets logged from the failed upload using the computer app

Debug webdav

Exception: {"Message":"HTTP/1.1 404 File with name .somefile.backup could not be located","Exception":"Sabre\DAV\Exception\NotFound",
"Code":0,"Trace":"
#0 /home/user/public_html/owncloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php(79): OCA\DAV\Connector\Sabre\ObjectTree->getNodeForPath('.somefile.bac...', 0)
#1 [internal function]: Sabre\DAV\CorePlugin->httpGet(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
#2 /home/user/public_html/owncloud/3rdparty/sabre/event/lib/EventEmitterTrait.php(105): call_user_func_array(Array, Array)
#3 /home/user/public_html/owncloud/3rdparty/sabre/dav/lib/DAV/Server.php(459): Sabre\Event\EventEmitter->emit('method:GET', Array)
#4 /home/user/public_html/owncloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php(251): Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response), false)
#5 [internal function]: Sabre\DAV\CorePlugin->httpHead(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
#6 /home/user/public_html/owncloud/3rdparty/sabre/event/lib/EventEmitterTrait.php(105): call_user_func_array(Array, Array)
#7 /home/user/public_html/owncloud/3rdparty/sabre/dav/lib/DAV/Server.php(459): Sabre\Event\EventEmitter->emit('method:HEAD', Array)
#8 /home/user/public_html/owncloud/3rdparty/sabre/dav/lib/DAV/Server.php(248): Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
#9 /home/user/public_html/owncloud/apps/dav/appinfo/v1/webdav.php(55): Sabre\DAV\Server->exec()
#10 /home/user/public_html/owncloud/remote.php(138): require_once('/home/user/...')
#11 {main}","File":"/home/user/public_html/owncloud/apps/dav/lib/connector/sabre/objecttree.php","Line":168,"User":"admin"}
PVince81 commented 8 years ago

@giuppi this is a normal message. The Android app first checks whether the file exists by doing a PROPFIND. If the PROPFIND returns 404, then it knows that it can upload without conflict.

This doesn't look related to "server did not acknowledge the last chunk" which is a message from the desktop client, not mobile.

PVince81 commented 8 years ago

I'm closing this huge ticket for now.

If anyone sees this issue again, please raise a new ticket and post all possible details using the issue template: https://raw.githubusercontent.com/owncloud/core/master/issue_template.md But first, check the forums or http://owncloud.org/support because in most cases it is a problem in the server environment and not a bug. Also note that many of the actual bugs that used to cause this error message have already been fixed.

Please always open a new ticket, unless an existing one contains the same exception or log entries from your owncloud.log, in which case the cause is the same.

Thank you!

j4hangir commented 8 years ago

For anyone who might still be tackling with this issue, in our setup of ownCloud 9.1.0, the culprit was mod_security, which was filtering certain file extensions.

lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.