[Bug]: 🪲 Server side encryption does not encrypt files with S3 primary storage, related error messages on encrypt-all and scan:legacy-format 😢

asheroto commented 11 months ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

Preface

This issue is well reported and discussed around GitHub and the Nextcloud Community forum, but the closest Issue is #33371 but @szaimen said to open a new bug report for ongoing issues which is why I'm creating a new issue for this. I'm hoping this issue gets solved. It seems like Nextcloud has a lot of issues with encryption, especially with S3. Many of these issues were reported months or years ago. 😊

Maybe most people do not use S3 as their primary storage, I'm not sure? S3 is an inexpensive way to store lots of data, especially with providers like Wasabi because it's only $6/TB/month. Hoping that someone takes ownership of this situation, even if that just means preventing new installs from using it or adding something to the manual. 😊

Remaining optimistic!

Background

Using Wasabi as an S3-compataible provider for primary storage in Nextcloud.

After upgrading from Nextcloud 26 or 27, I started receiving errors related to encryption (#8349) and thought I was able to solve it by disabling encryption and re-enabling it, but unfortunately most files remained unencrypted. After hours of troubleshooting, I decided to just start over from scratch.

So I installed a clean Nextcloud instance with a new database, new Redis, everything. But as soon as I set up S3 and enabled server side encryption, I ran into trouble. Files will sync and upload to the S3 provider but are not encrypted, unless they are very small. For example, text files will upload and encrypt, but even an 8 MB zip file will not encrypt. Same result in web client and in Windows client.

Because there are many similar issues, and even with a fresh install issues exists, it seems that there are still some unresolved issues within the codebase when it comes to this type of setup: S3 + server side encryption.

Troubleshooting

After performing the Steps to reproduce and then experiencing the issue, I troubleshooted by running these commands, testing with file uploads after each command:

The logs only appear when running or after running the occ commands. If I do not run the commands, I do not see anything in the log files. The uploads complete as expected without any logging, but the files are not encrypted.

Steps I have tried to resolve:

Decrypt all files: sudo -u www-data php occ encryption:decrypt-all
Verify encryption disabled: sudo -u www-data php occ encryption:status
Disable encryption: sudo -u www-data php occ encryption:disable
Scan all files to ensure integrity: sudo -u www-data php occ files:scan --all
Uncheck the Encrypt the home storage option in Settings > Security
Wait a minute
Enable encryption: sudo -u www-data php occ encryption:disable
Check the Encrypt the home storage option in Settings > Security
Wait a minute
Ensure encryption is enabled: sudo -u www-data php occ encryption:status
Scan key storage format: occ encryption:scan:legacy-format
Migrate key storage format for good measure: sudo -u www-data php occ encryption:migrate-key-storage-format
Encrypt all files: sudo -u www-data php occ encryption:encrypt-all

After running that, all of the files are encrypted.... but no new files become encrypted.

Many people are simply disabling the "Default Encryption Module" but that is not a solution because it disables encryption by doing that.

One of the issues recommended to implement these:

'encryption.legacy_format_support' => false,
'encryption.key_storage_migrated' => false,

But I had no luck with those.

Steps to reproduce

Install Nextcloud (I've seen the issue on both the archive method setup and nextcloud-snap)
Attach to an S3-compatible provider (I am using Wasabi)
Enable Default Encryption Module in Apps
Enable Server Side Encryption and Encrypt the Home Storage (should be default I believe)
Upload a small text file
Confirm the text file is encrypted on Wasabi
Upload a 10 MB file
Confirm the file is not encrypted on Wasabi

Expected behavior

Expected behavior is for files to automatically encrypt when uploaded.

Installation method

Community Manual installation with Archive

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "trashbin_retention_obligation": "30, 90",
        "remember_login_cookie_lifetime": 1296000,
        "filelocking.enabled": true,
        "trusted_domains": [
            "localhost",
            "***REDACTED***.com",
        ],
        "lost_password_link": "disabled",
        "overwriteprotocol": "https",
        "htaccess.RewriteBase": "\/",
        "versions_retention_obligation": "7, 30",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "app_install_overwrite": [
            "limit_login_to_ip"
        ],
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "REDACTED.storage",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "s3.us-central-1.wasabisys.com",
                "port": 443,
                "use_ssl": true,
                "region": "us-central-1",
                "use_path_style": false,
                "verify_bucket_exists": false
            }
        },
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "maintenance": false,
        "theme": "",
        "simpleSignUpLink.shown": false,
        "default_phone_region": "us",
        "default_locale": "en_US",
        "default_timezone": "America\/Chicago",
        "defaultapp": "files",
        "loglevel": 2,
        "overwrite.cli.url": "https:\/\/REDACTED.com",
        "data-fingerprint": "***REDACTED***",
        "updater.secret": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "27.1.4.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - activity: 2.19.0
  - audioplayer: 3.4.0
  - bruteforcesettings: 2.7.0
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contactsinteraction: 1.8.0
  - dashboard: 7.7.0
  - dav: 1.27.0
  - encryption: 2.15.0
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_external: 1.19.0
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - limit_login_to_ip: 4.0.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - nextcloud_announcements: 1.16.0
  - notifications: 2.15.0
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - support: 1.10.0
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0
  - text: 3.8.0
  - theming: 2.2.0
  - theming_customcss: 1.15.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_totp: 9.0.0
  - unsplash: 2.2.1
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - admin_audit: 1.17.0
  - geoblocker: 0.5.11 (installed 0.5.11)
  - survey_client: 1.15.0 (installed 1.15.0)
  - user_ldap: 1.17.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Will post below

Additional info

No response

asheroto commented 11 months ago

Other software info

php -m

``` [PHP Modules] apcu bcmath bz2 calendar Core ctype curl date dom exif FFI fileinfo filter ftp gd gettext gmp hash iconv igbinary imagick intl json libxml mbstring mysqli mysqlnd openssl pcntl pcre PDO pdo_mysql Phar posix random readline redis Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer xml xmlreader xmlwriter xsl Zend OPcache zip zlib [Zend Modules] Zend OPcache ```

php configuration

``` mbstring.func_overload=0 always_populate_raw_post_data=-1 default_charset='UTF-8' output_buffering=0 memory_limit=640M upload_max_filesize=16G post_max_size=16G max_input_time=3600 max_execution_time=3600 opcache.save_comments=1 opcache.revalidate_freq=60 opcache.validate_timestamps=0 opcache.jit=1255 opcache.jit_buffer_size=128M opcache.memory_consumption=256 opcache.interned_strings_buffer=128 ```

MySQL is configured per the recommended settings.

Log info

The first set of log messages only appear when running occ encryption:encrypt-all. I do not see any errors at all when uploading.

The second set of log messages only appear when running occ encryption:scan:legacy-format. I have implemented the OpenSSL workaround as mentioned in #32003 but that did not fix the issue for me even after restarting the php8.2-fpm and apache2 - the same error message appears.

Security & setup warnings

None

Summary

Honestly I'm not doing anything fancy. 😊 You can quickly replicate the problem for yourself if you simply perform the steps to reproduce. You can sign up with Wasabi for free in 5 minutes. It's usage-based billing. I installed a clean install of Nextcloud, php 8.2, Redis, MySQL, APCu, Apache. No service errors nor log errors from any of those where applicable.

As a side note, I was never able to get E2EE + SSE + S3 working. Only SSE + S3. But now that has seemed to stop as well unless manually engaged. :-/

I am hopeful for a resolution. I appreciate your time on this, thank you! 😁

Logs

logs.txt

Also

I also donate my time to many open source projects so I understand that time is valuable. So I really appreciate your help on this! I might be able to submit a PR but am unfamiliar with the inner-workings of Nextcloud from a developer standpoint. 😊

solracsf commented 11 months ago

I personally don't recommend using Nextcloud Encyption module with S3. Instead, use SSE-C with custom provided keys. https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html#s3-sse-c-encryption-support

asheroto commented 11 months ago

Thanks, I will try that as a workaround.

The disadvantage with that, of course, is that the files are transmitted unencrypted (encrypted in the connection, but unencrypted in memory). Although the S3 provider only stores the key in memory, it could be possible for bad actors within the S3 provider itself to intercept files or the key. This is of course highly unlikely but in theory possible, so is not the preferred method. This should work for now though until Nextcloud resolves the encryption issues with S3.

It seems that it is encrypting data but only when initiated manually through the occ encryption:encrypt-all command, so I think the bugs are pretty close to being resolved, I hope. 😊

Going to leave the ticket open since the issue itself is not resolved yet.

asheroto commented 11 months ago

I personally don't recommend using Nextcloud Encyption module with S3. Instead, use SSE-C with custom provided keys. https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html#s3-sse-c-encryption-support

Just tried this. Sadly another Nextcloud bug. Can we just use this issue to track the encryption issues? Seems to be all related to each other.

Could not decrypt key

Log errors for that: https://gist.github.com/asheroto/b4f13d528b48a304a0702d939c958494

Steps to reproduce: 1.) Use sse_c_key with Wasabi

Should be the same as AWS: https://knowledgebase.wasabi.com/hc/en-us/articles/4414850567963-How-does-SSE-C-Encryption-work-with-Wasabi-

This is sad that it doesn't work on a new installation. :-(

Let me know if I should create another ticket. All of these issues seem to be related.

asheroto commented 11 months ago

Oops, so when using SSE after having used encryption on Nextcloud, you have to keep the Default Encryption Module, but disable encryption. Otherwise you'll get that error.

Ideally the documentation should mention this, and instead of just saying Could not decrypt key it could help you and say Could not decrypt key. New SSE-C seen, try disabling encryption. or something like that. 😊

Thanks, will try this for now but hoping for a resolution on the main encryption method. 😊

asheroto commented 11 months ago

Possibly related to #41704. In reinstalling Nextcloud today (again), I found that error on a new installation.

nextcloud / server