[Bug]: Group Folders App is auto-disabled during upgrade and causes data loss

sonaro commented 1 year ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] Nextcloud Server is running on 64bit capable CPU, PHP and OS.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

During a major version upgrade, the Group Folders app was disabled for an unknown reason (needed an update?). While this app was disabled, and after Maintenance mode was turned off, any clients that had access to those group folders and that auto re-sync'd DELETED the client side group folders to match their new nextcloud server permissions since the GF app was disabled.

This is a major issue, for three primary reasons:

all of the group folders on the client, including any new unsync'd data saved to the client group folders were DELETED permanently and could not be recovered (unless the client had a local, separate backup running concurrently with nextcloud that also includes the Nextcloud local folder/group folders)
server-side backups were not able to recover lost data for any data that was not fully uploaded yet --- for instance, if files are written after maintenance started, or if quota was temporarily exceeded
wasted resources in having to re-download all of the content in the server group folders once the group folders issue was resolved

Upgrade history: -- 20.0.13 --> 20.0.14.2 --> 21.0.9.1 --> 22.2.10.2 --> 23.0.10.1 --> 24.0.6.1 -- backups were made between each version upgrade. Was not able to find the point when Group Folder app was disabled.

Steps to reproduce

This test plan also includes the client-side data loss example.

NC Client: 3.3.6 on OSX 12.4 (21F79) NC Server: 20.0.13, upgraded to 24.0.6.1 (Not sure when Group Folder app was disabled, but have all logs)

Turn on Maintenance Mode
Save a new test file to a group folder on a client filesystem that will be sync'd
Successfully update from a previous major release
Turn off Maintenance Mode
Check status of Group Folders App. If it says "Disabled", this bug applies
Wait for the client that no longer has access to the group folders to sync and it will delete the group folders, the group folder sync'd data, and and the unsync'd test file

To resolve access:

Group Folders app then needs to be manually updated and enabled via the web dashboard to restore access. Client data not sync'd cannot be restored (unless client has a separate concurrent backup)

Expected behavior

The expected behavior is to have the client side app "unmanage" the group folder and allow the entire folder to persist, rather than delete it.

Or:

"unmanage" and not delete any content in the group folder but never uploaded yet (only existing copy on the client side).
move the un-uploaded data to a different folder that the person can manually merge later on (not advised since client software may be saving to that location automatically)
at a minimum, immediate action could be to add a data loss warning any time the Group Folders app is disabled

Installation method

Community Web installer on a VPS or web space

Operating system

Debian/Ubuntu

PHP engine version

PHP 7.4

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 22.2.3 to 23.0.1)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.DOMAIN.TLD"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/cloud.DOMAIN.TLD",
        "dbtype": "mysql",
        "version": "24.0.6.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "dbindex": 0,
            "timeout": 1.5
        },
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpsecure": "ssl",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
        "theme": "",
        "loglevel": 0,
        "updater.release.channel": "stable",
        "mail_sendmailmode": "smtp",
        "app_install_overwrite": [
            "groupfolders",
            "passman"
        ],
        "encryption.legacy_format_support": false,
        "encryption.key_storage_migrated": false,
        "mysql.utf8mb4": true,
        "versions_retention_obligation": "auto",
        "twofactor_enforced": "true",
        "default_phone_region": "+1",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

NOTE:  this is the status of the system with "Group Folders" working properly (after the app was manually Enabled again)

Enabled:
  - accessibility: 1.10.0
  - activity: 2.16.0
  - admin_audit: 1.14.0
  - analytics: 4.4.0
  - bruteforcesettings: 2.4.0
  - calendar: 3.5.2
  - checksum: 1.1.5
  - circles: 24.0.1
  - cloud_federation_api: 1.7.0
  - comments: 1.14.0
  - contacts: 4.2.2
  - contactsinteraction: 1.5.0
  - dashboard: 7.4.0
  - dav: 1.22.0
  - drawio: 1.0.3
  - federatedfilesharing: 1.14.0
  - federation: 1.14.0
  - files: 1.19.0
  - files_external: 1.16.1
  - files_markdown: 2.3.6
  - files_pdfviewer: 2.5.0
  - files_rightclick: 1.3.0
  - files_sharing: 1.16.2
  - files_trashbin: 1.14.0
  - files_versions: 1.17.0
  - files_videoplayer: 1.13.0
  - firstrunwizard: 2.13.0
  - forms: 2.5.1
  - groupfolders: 12.0.2
  - logreader: 2.9.0
  - lookup_server_connector: 1.12.0
  - maps: 0.2.1
  - metadata: 0.17.0
  - nextcloud_announcements: 1.13.0
  - notes: 4.5.1
  - notifications: 2.12.1
  - oauth2: 1.12.0
  - onlyoffice: 7.5.6
  - ownpad: 0.6.18
  - password_policy: 1.14.0
  - photos: 1.6.0
  - privacy: 1.8.0
  - provisioning_api: 1.14.0
  - quota_warning: 1.15.0
  - recommendations: 1.3.0
  - serverinfo: 1.14.0
  - settings: 1.6.0
  - sharebymail: 1.14.0
  - support: 1.7.0
  - survey_client: 1.12.0
  - systemtags: 1.14.0
  - tasks: 0.14.5
  - text: 3.5.1
  - theming: 1.15.0
  - twofactor_backupcodes: 1.13.0
  - twofactor_totp: 6.4.1
  - unsplash: 2.0.1
  - updatenotification: 1.14.0
  - user_ldap: 1.14.1
  - user_status: 1.4.0
  - viewer: 1.8.0
  - weather_status: 1.4.0
  - workflowengine: 2.6.0
Disabled:
  - deck: 1.7.2
  - emlviewer: 1.0.2
  - encryption: 2.7.0
  - external: 4.0.0
  - files_accesscontrol: 1.14.1
  - files_automatedtagging: 1.14.0
  - limit_login_to_ip: 3.1.0
  - passman: 2.1.5
  - ransomware_detection: 0.10.0
  - richdocuments: 6.3.1
  - spreed: 14.0.5
  - twofactor_u2f: 6.3.1
  - user_saml: 5.0.3

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Logs are huge and also contain sensitive data.  Please ask and I can provide relevant sections.

Additional info

I'm not yet able to determine exactly when Group Folders app was disabled, so any info on how to best filter through the logs would be most helpful.

camilasan commented 1 year ago

Any chance https://github.com/nextcloud/groupfolders/issues/1816 is related?

sonaro commented 1 year ago

@camilasan https://github.com/nextcloud/groupfolders/issues/1816 is similar, but distinct. In this case, the group folder disappeared because of a server-side action (Group Folders app because disabled), not because of a client-side rename/delete/move.

sonaro commented 1 year ago

@szaimen I want to be careful assigning this to 24.x, since I'm not exactly sure during which major upgrade Group Folders app was disabled. (it happened somewhere 20.0.13 --> 20.0.14.2 --> 21.0.9.1 --> 22.2.10.2 --> 23.0.10.1 --> 24.0.6.1) Could someone advise me the where to look to determine how/when the Group Folders app was disabled?

simon2cyrene commented 1 year ago

For me, the Group Folders app was disabled while the 23.0.10 to 24.0.7 upgrade. Same result → the folders are not any more in the database. The data are not exactly loss, they are still present in file system.

diniciacci commented 1 year ago

I can confirm the original poster issue. We updated 22->23->24 and all was fine, 24->25 gave the exact same issue as reported. Data loss on all desktop-clients both on windows and on linux.

We are sure the directories have not been renamed by users. Just on windows clients a directory named "\<original name> (1)" has been created for each group folder automatically by the desktop client.

mirkowi commented 1 year ago

I confirm this issue. With update to Nextcloud Hub 4 (26.0.0) the Group Folders App was disabled and all client data loss. After enabling the App the configuration is empty.

provokateurin commented 1 month ago

Wrong transfer :sweat:

provokateurin commented 1 month ago

As for the Desktop client, it is doing everything right.

I don't know why the app got disabled automatically, but the admin UI tells you what apps will be unavailable after the update, so you should check that before updating.

Please re-open if there is any information on when exactly it went wrong, as nobody seems to be able to pin down the exact version.

nextcloud / groupfolders