end2end encryption - huge performance impact when only a single folder is synchronised and encrypted

Steps to reproduce

everytime when enabling e2ee and syching / encrypting some folders.

Expected behaviour

smooth transactions, no errors at all, ho heavy impact to server cpu, memory, mysql load. Sync process together with encryption needs more ressources, that's clear & ok. But one single sync/encryption process cannot kill the server.

Actual behaviour

I did encrypt only one single folder with a few files/Mbytes ..

I've got a powerful Enterprise Server with several vCPUs, enough memory .. but by starting the Desktop agent or Encrypting such a Folder (only a few files with a few Mbytes!) .. the VM gets much load on CPU (all vCUPs over 50%) and the memory consumption increases also much.

mysql and php-fpm processes now require almost all resources during encryption or agent initialization & and this, as already mentioned, with a very powerful server.

I wonder if something needs to be improved in design ... because what happens when 2,5,10 or even 100 users encrypt some of their folders ?

The Desktop Agents now needs a huge time to initialize - also when changes happen (changed files) on the Desktop side.

i did increase PHP Memory size now to 1GB upon those error messages from PHP -->

[24-Oct-2020 03:31:15 UTC] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 158 [24-Oct-2020 03:33:11 UTC] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 158 [24-Oct-2020 05:34:40 UTC] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/nextcloud/lib/private/Files/Cache/Cache.php on line 175

I know, there is a similar issue -->

https://github.com/nextcloud/desktop/issues/2519#issuecomment-683272921

This case is a bit different in this sense, as it is a fresh start for e2ee and especially - because e2ee should be Production ready - but it is not in my eyes. I'm sorry to say that - I had already sent a newsletter to my users ... but it's really hard for me to say that I have to write a new newsletter immediately and explain my useres,why e2ee is not yet production ready. I'm an absolute Nextcloud fan and the whole Nextcloud team is just great - but this time they were probably too fast and too opitmistic with e2ee. Encryption is one of the most important milestones and has to be handled very carefully to avoid any disaster.

Server configuration detail

Operating system: Linux 4.18.0-193.19.1.el8_2.x86_64 nextcloud/end_to_end_encryption#1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64

Webserver: Apache (fpm-fcgi)

Database: mysql 10.3.17

PHP version:

7.4.11 Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, cgi-fcgi, bcmath, bz2, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, gmp, iconv, intl, json, ldap, exif, mysqlnd, PDO, Phar, posix, shmop, SimpleXML, sockets, sodium, sqlite3, sysvmsg, sysvsem, sysvshm, tokenizer, xml, xmlwriter, xsl, mysqli, pdo_mysql, pdo_sqlite, xmlreader, apcu, igbinary, imagick, msgpack, smbclient, zip, memcached, redis, libsmbclient, Zend OPcache

Nextcloud version: 19.0.4 - 19.0.4.2

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: unknown

Signing status

Array ( )

List of activated apps

``` Enabled: - accessibility: 1.5.0 - activity: 2.12.1 - analytics: 2.5.0 - announcementcenter: 3.8.1 - apporder: 0.11.0 - audioplayer: 2.12.0 - audioplayer_editor: 0.3.0 - audioplayer_sonos: 1.2.0 - bookmarks: 3.4.4 - bruteforcesettings: 2.0.1 - calendar: 2.0.4 - circles: 0.19.7 - cloud_federation_api: 1.2.0 - comments: 1.9.0 - contacts: 3.4.1 - contactsinteraction: 1.0.0 - cookbook: 0.7.6 - cospend: 1.0.5 - data_request: 1.6.0 - dav: 1.15.0 - deck: 1.1.2 - dicomviewer: 1.2.2 - documentserver_community: 0.1.8 - drawio: 0.9.7 - event_update_notification: 1.0.2 - extract: 1.2.4 - federatedfilesharing: 1.9.0 - federation: 1.9.0 - files: 1.14.0 - files_3d: 0.3.2 - files_antivirus: 3.0.0 - files_automatedtagging: 1.9.0 - files_downloadactivity: 1.8.0 - files_markdown: 2.3.1 - files_mindmap: 0.0.23 - files_pdfviewer: 1.8.0 - files_photospheres: 1.19.1 - files_rightclick: 0.16.0 - files_sharing: 1.11.0 - files_trashbin: 1.9.0 - files_versions: 1.12.0 - files_videoplayer: 1.8.0 - firstrunwizard: 2.8.0 - flowupload: 1.0.0 - forms: 2.0.4 - gpxedit: 0.0.13 - gpxmotion: 0.0.11 - gpxpod: 4.2.2 - groupfolders: 7.1.0 - impersonate: 1.6.1 - issuetemplate: 0.7.0 - logreader: 2.4.0 - lookup_server_connector: 1.7.0 - maps: 0.1.6 - nextcloud_announcements: 1.8.0 - notes: 3.6.4 - notifications: 2.7.0 - oauth2: 1.7.0 - onlyoffice: 6.1.0 - passman: 2.3.6 - password_policy: 1.9.1 - photos: 1.1.0 - polls: 1.5.4 - privacy: 1.3.0 - provisioning_api: 1.9.0 - quicknotes: 0.6.1 - quota_warning: 1.8.0 - rainloop: 7.0.3 - ransomware_protection: 1.7.0 - recommendations: 0.7.0 - serverinfo: 1.9.0 - settings: 1.1.0 - sharebymail: 1.9.0 - socialsharing_diaspora: 2.1.0 - socialsharing_email: 2.1.0 - socialsharing_facebook: 2.1.0 - socialsharing_twitter: 2.1.0 - spreed: 9.0.5 - support: 1.2.1 - suspicious_login: 3.2.1 - systemtags: 1.9.0 - tasks: 0.13.5 - terms_of_service: 1.5.2 - text: 3.0.1 - theming: 1.10.0 - timemanager: 0.1.4 - twofactor_backupcodes: 1.8.0 - twofactor_totp: 5.0.0 - updatenotification: 1.9.0 - video_converter: 0.1.5 - viewer: 1.3.0 - workflow_ocr: 1.19.1 - workflow_pdf_converter: 1.4.0 - workflow_script: 1.4.0 - workflowengine: 2.1.0 Disabled: - admin_audit - breezedark - dashboard - drop_account - encryption - end_to_end_encryption - external - files_accesscontrol - files_external - files_fulltextsearch - fulltextsearch - fulltextsearch_elasticsearch - joplin - jsloader - passwords - registration - richdocuments - socialsharing_googleplus - survey_client - user_ldap - weather ```

Configuration (config/config.php)

``` { "memcache.local": "\\OC\\Memcache\\APCu", "filelocking.enabled": true, "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": "***REMOVED SENSITIVE VALUE***",, "dbindex": "***REMOVED SENSITIVE VALUE***",, "timeout": 1.5, "password": "***REMOVED SENSITIVE VALUE***" }, "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ ""***REMOVED SENSITIVE VALUE***",", ""***REMOVED SENSITIVE VALUE***"," ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": ""***REMOVED SENSITIVE VALUE***", "htaccess.RewriteBase": "\/", "overwriteprotocol": "https", "dbtype": "mysql", "version": "19.0.4.2", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "maintenance": false, "theme": "", "loglevel": 0, "updater.release.channel": "stable", "auth.bruteforce.protection.enabled": true, "check_for_working_htaccess": true, "data-fingerprint": ""***REMOVED SENSITIVE VALUE***",", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_smtpauthtype": "LOGIN", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtpsecure": "tls", "mail_smtpauth": 1, "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "587", "session_lifetime": 1200, "session_keepalive": false, "logtimezone": ""***REMOVED SENSITIVE VALUE***",", "logfile": "\/media\/log\/nextcloud.log", "knowledgebaseenabled": false, "log_rotate_size": 3145728, "mail_sendmailmode": "smtp", "app_install_overwrite": [ "passman", "dicomviewer", "radio" ] } ```

Are you using external storage, if yes which one: no

Are you using encryption:

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36 Edg/86.0.622.51

Operating system: Windows 10 2004

Logs

Web server error log

``` ```

Nextcloud log

``` ```

Browser log

nextcloud / desktop