ferdiga
commented
3 years ago some additional information
- it's Sunday, the serverload of the target NC virtual machine is next to 0
- the source directory is NOT a group folder
- the move of 15GB directories works like charm
- the user on target NC has not many data and the target directory (federation) was always emptied before moving.
provokateurin
Steps to reproduce
move large folders - typically 5GB and more with a some 1000 files to a federated folder
Expected behaviour
move all files (or none - rollback)
Actual behaviour
after some time - but not always - the process crashes "can't move ...." It happens many times, but I can not detect a pattern. guesswork here: It looks like a timeout (php timeout), but it runs definitely longer than the typically 60 seconds) it seems that there are concurrent transfer processes running and and started at different point of times. So it could be a timeout of one of these later started processes
after crash: no copy, move, download operations are possible
source DB
target DB - not all files arrived BTW I uses webdav mounted shares to rsync source and target it would be great if such a tool could be provided as app the NC (at least as long this process is not stable)
I suspect that the crash interrupts the process which updates the database - hence the database does not reflect the actual state of the files any more and this leads at least to data inconsistency.
Server configuration detail
Operating system: Linux 4.19.0-10-amd64 nextcloud/server#1 SMP Debian 4.19.132-1 (2020-07-24) x86_64
Webserver: Apache/2.4.38 (Debian) (apache2handler)
Database: mysql 10.3.25
PHP version:
7.3.19-1~deb10u1 Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, sodium, standard, apache2handler, mysqlnd, PDO, xml, apcu, bcmath, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv, imagick, intl, json, ldap, exif, mysqli, pdo_mysql, pdo_pgsql, pdo_sqlite, pgsql, apc, posix, readline, shmop, SimpleXML, sockets, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Phar, Zend OPcache
Nextcloud version: 20.0.8 - 20.0.8.1
Updated from an older Nextcloud/ownCloud or fresh install:
Where did you install Nextcloud from: unknown
Signing status
Array ( )List of activated apps
``` Enabled: - accessibility: 1.6.0 - activity: 2.13.4 - admin_audit: 1.10.0 - analytics: 3.4.1 - announcementcenter: 4.0.1 - apporder: 0.12.0 - audioplayer: 3.1.0 - bruteforcesettings: 2.0.1 - calendar: 2.2.0 - carnet: 0.23.8 - circles: 0.20.7 - cloud_federation_api: 1.3.0 - comments: 1.10.0 - contacts: 3.5.1 - contactsinteraction: 1.1.0 - dashboard: 7.0.0 - dav: 1.16.2 - deck: 1.2.5 - external: 3.7.2 - federatedfilesharing: 1.10.2 - federation: 1.10.1 - files: 1.15.0 - files_external: 1.11.1 - files_external_gdrive: 0.4.0 - files_pdfviewer: 2.0.1 - files_photospheres: 1.20.1 - files_rightclick: 0.17.0 - files_sharing: 1.12.2 - files_trashbin: 1.10.1 - files_versions: 1.13.0 - files_videoplayer: 1.9.0 - groupfolders: 8.2.0 - integration_google: 1.0.0 - issuetemplate: 0.7.0 - logreader: 2.5.0 - lookup_server_connector: 1.8.0 - metadata: 0.13.0 - nextcloud_announcements: 1.9.0 - notes: 4.0.4 - notifications: 2.8.0 - oauth2: 1.8.0 - ocr: 6.0.58 - password_policy: 1.10.1 - phonetrack: 0.6.7 - photos: 1.2.3 - previewgenerator: 3.1.1 - privacy: 1.4.0 - provisioning_api: 1.10.0 - quickaccesssorting: 1.1.2 - quicknotes: 0.6.7 - quota_warning: 1.9.1 - rainloop: 7.1.2 - recommendations: 0.8.0 - richdocuments: 3.7.17 - richdocumentscode: 6.4.705 - serverinfo: 1.10.0 - settings: 1.2.0 - sharebymail: 1.10.0 - socialsharing_email: 2.1.0 - spreed: 10.0.6 - support: 1.3.0 - survey_client: 1.8.0 - suspicious_login: 3.2.1 - systemtags: 1.10.0 - tasks: 0.13.6 - text: 3.1.0 - theming: 1.11.0 - theming_customcss: 1.8.0 - twofactor_backupcodes: 1.9.0 - updatenotification: 1.10.0 - user_status: 1.0.1 - user_usage_report: 1.4.2 - viewer: 1.4.0 - weather_status: 1.0.0 - whiteboard: 0.0.3 - workflow_ocr: 1.20.2 - workflow_pdf_converter: 1.5.1 - workflow_script: 1.5.1 - workflowengine: 2.2.0 Disabled: - dashboardcharts - drawio - emlviewer - encryption - files_accesscontrol - files_automatedtagging - files_downloadactivity - files_fulltextsearch - files_fulltextsearch_tesseract - files_retention - firstrunwizard - fulltextsearch - fulltextsearch_elasticsearch - mail - ocsms - ojsxc - printer - registration - user_ldap - wopi ```Configuration (config/config.php)
``` { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "20.0.8.1", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "debug": false, "installed": true, "enable_previews": true, "forcessl": true, "theme": "", "maintenance": false, "mail_smtpmode": "smtp", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "587", "mail_smtptimeout": 10, "loglevel": 2, "allow_local_remote_servers": true, "share_folder": "\/Shared", "trusted_domains": [ "service.chricar.at" ], "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "singleuser": false, "forceSSLforSubdomains": true, "trashbin_retention_obligation": "auto", "filelocking.enabled": "true", "filesystem_check_changes": 1, "mail_smtpauthtype": "PLAIN", "mail_smtpsecure": "tls", "mail_smtpauth": 1, "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "auth.bruteforce.protection.enabled": true, "overwriteprotocol": "https", "overwrite.cli.url": "https:\/\/service.chricar.at\/owncloud", "updater.release.channel": "stable", "memcache.local": "\\OC\\Memcache\\APCu", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 6379, "password": "***REMOVED SENSITIVE VALUE***", "timeout": 1.5 }, "preview_max_x": 512, "preview_max_y": 512, "preview_max_scale_factor": 1, "preview_libreoffice_path": "\/usr\/bin\/libreoffice", "enabledPreviewProviders": [ "OC\\Preview\\Image", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\PDF", "OC\\Preview\\Movie", "OC\\Preview\\Photoshop", "OC\\Preview\\TIFF", "OC\\Preview\\SVG", "OC\\Preview\\OpenDocument" ], "has_rebuilt_cache": true, "mysql.utf8mb4": true, "app_install_overwrite": [ "files_clipboard", "files_accesscontrol", "issuetemplate", "dashboard", "files_external_gdrive", "ojsxc", "quickaccesssorting", "ocr", "files_fulltextsearch_tesseract" ], "0": { "verify_peer_off": true }, "data-fingerprint": "2e5954a0f3c41d5dc2c0cd22b19dc1e0" } ```Are you using external storage, if yes which one: local/smb/sftp/...
Are you using encryption:
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
Client configuration
Browser: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36
Operating system:
Logs
Web server error log
``` Insert your web server log here ```Nextcloud log
``` Insert your Nextcloud log here many many like Fatal webdav Sabre\DAV\Exception\ServiceUnavailable: Storage is temporarily not available Error no app in context OCP\Files\StorageNotAvailableException: Fatal webdav Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'UPDATE `oc_filecache` SET `storage` = ?, `path` = ?, `path_hash` = ?, `name` = ?, `parent` = ? WHERE `fileid` = ?' with params [46, "__groupfolders\/trash\/8\/Inventory.d1616786381", "cff8cb1fd8a18fced7f4614b249f3fce", "Inventory.d1616786381", 43818419, 36696734]: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '46-cff8cb1fd8a18fced7f4614b249f3fce' for key 'fs_storage_path_hash'x Warning core Login failed: 'me' (Remote IP: 'xxx.20.xxx.250') (my clients address) ```Browser log
Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ...