nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
27.22k stars 4.05k forks source link

Error downloading or uploading large files #9801

Closed ludovi-com closed 6 years ago

ludovi-com commented 6 years ago

Steps to reproduce

Sometimes, randomly, downloading or uploading large files

Expected behaviour

Synchronise the file

Actual behaviour

Skip the file and returns error : Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTO oc_file_locks (key,lock,ttl) SELECT ?,?,? FROM oc_file_locks WHERE key = ? HAVING COUNT(*) = 0' with params ["files\/b8d1b7bbcdb48962b43c548833754976", 1, 1528461715, "files\/b8d1b7bbcdb48962b43c548833754976"]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files/b8d1b7bbcdb48962b43c548833754976' for key 'lock_key_index'

Server configuration

Operating system: CentOS Linux release 7.5.1804 (Core)

Web server: Apache (httpd-2.4.6-80.el7.centos.x86_64)

Database: Ver 8.0.11 for Linux on x86_64 (MySQL Community Server - GPL)

PHP version: PHP 7.2.6

Nextcloud version: 13.0.3

Updated from an older Nextcloud/ownCloud or fresh install: Fresh install

Where did you install Nextcloud from: Download from NextCloud website

Signing status:

Signing status ``` No errors have been found. ```

List of activated apps:

App list ``` Enabled: - activity: 2.6.1 - bruteforcesettings: 1.0.3 - comments: 1.3.0 - dav: 1.4.7 - federatedfilesharing: 1.3.1 - federation: 1.3.0 - files: 1.8.0 - files_sharing: 1.5.0 - files_texteditor: 2.5.1 - files_trashbin: 1.3.0 - files_versions: 1.6.0 - files_videoplayer: 1.2.0 - firstrunwizard: 2.2.1 - gallery: 18.0.0 - logreader: 2.0.0 - lookup_server_connector: 1.1.0 - nextcloud_announcements: 1.2.0 - notifications: 2.1.2 - oauth2: 1.1.1 - password_policy: 1.3.0 - provisioning_api: 1.3.0 - serverinfo: 1.3.0 - sharebymail: 1.3.0 - survey_client: 1.1.0 - systemtags: 1.3.0 - theming: 1.4.5 - twofactor_backupcodes: 1.2.3 - updatenotification: 1.3.0 - workflowengine: 1.3.0 Disabled: - admin_audit - encryption - files_external - files_pdfviewer - user_external - user_ldap ```

Nextcloud configuration:

Config report ``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "***", "dbtype": "mysql", "version": "13.0.3.2", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "mail_smtpmode": "php", "mail_smtpauthtype": "LOGIN", "mail_smtpsecure": "ssl", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "htaccess.RewriteBase": "\/nextcloud", "memcache.local": "\\OC\\Memcache\\APCu", "enable_previews": true, "preview_max_x": 1280, "preview_max_y": 720, "enabledPreviewProviders": [ "OC\\Preview\\Movie", "OC\\Preview\\Image" ], "enable_movie_transcode": true } } ```

Are you using external storage, if yes which one: local (Setting data folder to a binded mount point)

Are you using encryption: No

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

LDAP configuration (delete this part if not used)

LDAP config ``` [Symfony\Component\Console\Exception\CommandNotFoundException] There are no commands defined in the "ldap" namespace. ```

Client configuration

Browser: Error encountered in various browsers (Safari, Firefox, Chrome) and desktop clients

Operating system: Error encountered in various Operating Systems (Windows 10 and MacOS 10.13)

Logs

Web server error log

Web server error log ``` Insert your webserver log here ```

Nextcloud log (data/nextcloud.log)

Nextcloud log ``` {"reqId":"WxmuIkfOtVyixusuLkozrQAAAAk","level":2,"time":"2018-06-07T22:13:54+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"--","app":"core","method":"POST","url":"\/nextcloud\/login","message":"Login failed: 'ludovico' (Remote IP: '77.43.1.139')","userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/605.1.15 (KHTML, like Gecko) Version\/11.1 Safari\/605.1.15","version":"13.0.3.2"} {"reqId":"WxoNXmWYmFj6zHmZkfo4wgAAAAg","level":3,"time":"2018-06-08T05:00:15+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"GET","url":"\/nextcloud\/apps\/files_trashbin\/ajax\/preview.php?file=%2FNextcloud.mp4.d1528401419&c=1528401419000","message":"unlink(\/mnt\/data\/php_tmp\/oc_tmp_wIVnIZ): No such file or directory at \/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Movie.php#111","userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit\/605.1.15 (KHTML, like Gecko) Version\/11.1.1 Safari\/605.1.15","version":"13.0.3.2"} {"reqId":"WxoNaGNy7xdk-yjksn7tzwAAAAc","level":3,"time":"2018-06-08T05:00:24+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"POST","url":"\/nextcloud\/index.php\/apps\/files_trashbin\/ajax\/delete.php","message":"filesize(): stat failed for \/var\/www\/html\/nextdata\/nextdata\/\/ludovico\/files_trashbin\/files\/FILE019.MOV.d1528409613 at \/var\/www\/html\/nextcloud\/lib\/private\/Files\/Storage\/Local.php#166","userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit\/605.1.15 (KHTML, like Gecko) Version\/11.1.1 Safari\/605.1.15","version":"13.0.3.2"} {"reqId":"WxoNaGNy7xdk-yjksn7tzwAAAAc","level":3,"time":"2018-06-08T05:00:24+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"POST","url":"\/nextcloud\/index.php\/apps\/files_trashbin\/ajax\/delete.php","message":"filesize(): stat failed for \/var\/www\/html\/nextcloud\/nextdata\/nextdata\/\/ludovico\/files_trashbin\/files\/FILE019.MOV.d1528401438 at \/var\/www\/html\/nextcloud\/lib\/private\/Files\/Storage\/Local.php#166","userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit\/605.1.15 (KHTML, like Gecko) Version\/11.1.1 Safari\/605.1.15","version":"13.0.3.2"} {"reqId":"WxoNg3QiHCnVq7u2m5QC@wAAAAU","level":2,"time":"2018-06-08T05:00:51+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"--","app":"core","method":"POST","url":"\/nextcloud\/login","message":"Login failed: 'root' (Remote IP: '***REMOVED SENSITIVE VALUE***')","userAgent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit\/605.1.15 (KHTML, like Gecko) Version\/11.1.1 Safari\/605.1.15","version":"13.0.3.2"} {"reqId":"Wxo0c7bjqnIscLQ9skMsJgAAAAQ","level":3,"time":"2018-06-08T07:47:00+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"francesco","app":"PHP","method":"GET","url":"\/nextcloud\/core\/preview.png?file=%2FNextcloud.mp4&c=1f631ae1ed17006fe4bff343c746a370&x=56&y=56&forceIcon=0","message":"unlink(\/mnt\/data\/php_tmp\/oc_tmp_W6wM1W): No such file or directory at \/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Movie.php#111","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.79 Safari\/537.36","version":"13.0.3.2"} {"reqId":"Wxo1MfF5dfWvvx8bPPbq4gAAAAY","level":3,"time":"2018-06-08T07:50:25+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"GET","url":"\/nextcloud\/index.php\/apps\/files\/api\/v1\/thumbnail\/128\/128\/Drone\/18-05-26\/DJI_0193.MOV","message":"unlink(\/mnt\/data\/php_tmp\/oc_tmp_vXDTcD): No such file or directory at \/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Movie.php#111","userAgent":"Mozilla\/5.0 (iOS) Nextcloud-iOS\/2.21.0","version":"13.0.3.2"} {"reqId":"Wxo1MfF5dfWvvx8bPPbq4gAAAAY","level":3,"time":"2018-06-08T07:50:40+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"GET","url":"\/nextcloud\/index.php\/apps\/files\/api\/v1\/thumbnail\/128\/128\/Drone\/18-05-26\/DJI_0193.MOV","message":"unlink(\/mnt\/data\/php_tmp\/oc_tmp_7WOse0): No such file or directory at \/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Movie.php#111","userAgent":"Mozilla\/5.0 (iOS) Nextcloud-iOS\/2.21.0","version":"13.0.3.2"} {"reqId":"Wxo1RMMLUoGt9@8dbOYoSQAAAAc","level":3,"time":"2018-06-08T07:50:42+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"GET","url":"\/nextcloud\/index.php\/apps\/files\/api\/v1\/thumbnail\/128\/128\/Drone\/18-05-26\/DJI_0193.MOV","message":"unlink(\/mnt\/data\/php_tmp\/oc_tmp_FQxLfU): No such file or directory at \/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Movie.php#111","userAgent":"Mozilla\/5.0 (iOS) Nextcloud-iOS\/2.21.0","version":"13.0.3.2"} {"reqId":"Wxo1QzZM7gZEXNSS1id0rQAAAAA","level":3,"time":"2018-06-08T07:50:43+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"GET","url":"\/nextcloud\/index.php\/apps\/files\/api\/v1\/thumbnail\/128\/128\/Drone\/18-05-26\/DJI_0193.MOV","message":"unlink(\/mnt\/data\/php_tmp\/oc_tmp_JmDp8O): No such file or directory at \/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Movie.php#111","userAgent":"Mozilla\/5.0 (iOS) Nextcloud-iOS\/2.21.0","version":"13.0.3.2"} {"reqId":"Wxo1RMMLUoGt9@8dbOYoSQAAAAc","level":3,"time":"2018-06-08T07:50:54+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"GET","url":"\/nextcloud\/index.php\/apps\/files\/api\/v1\/thumbnail\/128\/128\/Drone\/18-05-26\/DJI_0193.MOV","message":"unlink(\/mnt\/data\/php_tmp\/oc_tmp_DVUFf3): No such file or directory at \/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Movie.php#111","userAgent":"Mozilla\/5.0 (iOS) Nextcloud-iOS\/2.21.0","version":"13.0.3.2"} {"reqId":"Wxo1QzZM7gZEXNSS1id0rQAAAAA","level":3,"time":"2018-06-08T07:51:05+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"GET","url":"\/nextcloud\/index.php\/apps\/files\/api\/v1\/thumbnail\/128\/128\/Drone\/18-05-26\/DJI_0193.MOV","message":"unlink(\/mnt\/data\/php_tmp\/oc_tmp_LS2e1Z): No such file or directory at \/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Movie.php#111","userAgent":"Mozilla\/5.0 (iOS) Nextcloud-iOS\/2.21.0","version":"13.0.3.2"} {"reqId":"Wxo1YSd0amOCaDOaaR-EnwAAAAs","level":3,"time":"2018-06-08T07:51:12+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"GET","url":"\/nextcloud\/index.php\/apps\/files\/api\/v1\/thumbnail\/128\/128\/Drone\/18-05-26\/DJI_0193.MOV","message":"unlink(\/mnt\/data\/php_tmp\/oc_tmp_amGoXN): No such file or directory at \/var\/www\/html\/nextcloud\/lib\/private\/Preview\/Movie.php#111","userAgent":"Mozilla\/5.0 (iOS) Nextcloud-iOS\/2.21.0","version":"13.0.3.2"} {"reqId":"Wxo1YSd0amOCaDOaaR-EnwAAAAs","level":3,"time":"2018-06-08T07:51:31+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"PHP","method":"GET","url":"\/nextcloud\/index.php\/apps\/files\/api\/v1\/thumbnail\/128\/128\/Drone\/18-05-26\/DJI_0193.MOV","message":"unlink(\/mnt\/data\/php_tmp\/oc_tmp_2AU5Jj): No such file or directory at \/var\/www\/html\/lib\/private\/Preview\/Movie.php#111","userAgent":"Mozilla\/5.0 (iOS) Nextcloud-iOS\/2.21.0","version":"13.0.3.2"} {"reqId":"Wxprg10xFcHfuwCM62iX-gAAAAE","level":4,"time":"2018-06-08T11:41:55+00:00","remoteAddr":"***REMOVED SENSITIVE VALUE***","user":"ludovico","app":"webdav","method":"GET","url":"\/nextcloud\/remote.php\/dav\/files\/Drone\/18-05-18\/FILE002.MOV","message":"Exception: {\"Exception\":\"Doctrine\\\\DBAL\\\\Exception\\\\UniqueConstraintViolationException\",\"Message\":\"An exception occurred while executing 'INSERT INTO `oc_file_locks` (`key`,`lock`,`ttl`) SELECT ?,?,? FROM `oc_file_locks` WHERE `key` = ? HAVING COUNT(*) = 0' with params [\\\"files\\\\\\\/b8d1b7bbcdb48962b43c548833754976\\\", 1, 1528461715, \\\"files\\\\\\\/b8d1b7bbcdb48962b43c548833754976\\\"]:\\n\\nSQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'files\\\/b8d1b7bbcdb48962b43c548833754976' for key 'lock_key_index'\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/doctrine\\\/dbal\\\/lib\\\/Doctrine\\\/DBAL\\\/DBALException.php(128): Doctrine\\\\DBAL\\\\Driver\\\\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\\\\DBAL\\\\Driver\\\\PDOException))\\n#1 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/doctrine\\\/dbal\\\/lib\\\/Doctrine\\\/DBAL\\\/Connection.php(1015): Doctrine\\\\DBAL\\\\DBALException::driverExceptionDuringQuery(Object(Doctrine\\\\DBAL\\\\Driver\\\\PDOMySql\\\\Driver), Object(Doctrine\\\\DBAL\\\\Driver\\\\PDOException), 'INSERT INTO `oc...', Array)\\n#2 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/DB\\\/Connection.php(216): Doctrine\\\\DBAL\\\\Connection->executeUpdate('INSERT INTO `oc...', Array, Array)\\n#3 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/DB\\\/Adapter.php(114): OC\\\\DB\\\\Connection->executeUpdate('INSERT INTO `oc...', Array)\\n#4 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/DB\\\/Connection.php(254): OC\\\\DB\\\\Adapter->insertIfNotExist('*PREFIX*file_lo...', Array, Array)\\n#5 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/Lock\\\/DBLockingProvider.php(119): OC\\\\DB\\\\Connection->insertIfNotExist('*PREFIX*file_lo...', Array, Array)\\n#6 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/Lock\\\/DBLockingProvider.php(164): OC\\\\Lock\\\\DBLockingProvider->initLockField('files\\\/b8d1b7bbc...', 1)\\n#7 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Common.php(709): OC\\\\Lock\\\\DBLockingProvider->acquireLock('files\\\/b8d1b7bbc...', 1)\\n#8 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(592): OC\\\\Files\\\\Storage\\\\Common->acquireLock('files\\\/Drone', 1, Object(OC\\\\Lock\\\\DBLockingProvider))\\n#9 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(1924): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->acquireLock('files\\\/Drone', 1, Object(OC\\\\Lock\\\\DBLockingProvider))\\n#10 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(2038): OC\\\\Files\\\\View->lockPath('\\\/Drone', 1)\\n#11 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(1134): OC\\\\Files\\\\View->lockFile('\\\/Drone\\\/18-05-18...', 1)\\n#12 \\\/var\\\/www\\\/html\\\/nextcloud\\\/lib\\\/private\\\/Files\\\/View.php(987): OC\\\\Files\\\\View->basicOperation('fopen', '\\\/Drone\\\/18-05-18...', Array, 'r')\\n#13 \\\/var\\\/www\\\/html\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php(335): OC\\\\Files\\\\View->fopen('Drone\\\/18-05-18\\\/...', 'r')\\n#14 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(88): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->get()\\n#15 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpGet(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#16 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#17 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:GET', Array)\\n#18 \\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#19 \\\/var\\\/www\\\/html\\\/nextcloud\\\/apps\\\/dav\\\/lib\\\/Server.php(287): Sabre\\\\DAV\\\\Server->exec()\\n#20 \\\/var\\\/www\\\/html\\\/nextcloud\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(35): OCA\\\\DAV\\\\Server->exec()\\n#21 \\\/var\\\/www\\\/html\\\/nextcloud\\\/remote.php(164): require_once('\\\/var\\\/www\\\/html\\\/n...')\\n#22 {main}\",\"File\":\"\\\/var\\\/www\\\/html\\\/nextcloud\\\/3rdparty\\\/doctrine\\\/dbal\\\/lib\\\/Doctrine\\\/DBAL\\\/Driver\\\/AbstractMySQLDriver.php\",\"Line\":66}","userAgent":"Mozilla\/5.0 (Macintosh) mirall\/2.3.3 (build 84) (Nextcloud)","version":"13.0.3.2"} ```

Browser log

Browser log ``` Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ... ```
MorrisJobke commented 6 years ago

In more recent versions (13.0.4 and 12.0.9) we reduced drastically the amount of lock statements which should avoid most of the problems in here. Thus I will close it. If there is still stuff that breaks have a look at #9305 and it's linked issues.

Another workaround: use Redis as locking backend