search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.23k stars 3.95k forks source link

[Bug]: Folders sometimes empty after upgrade to NC29 #46098

Open paramazo opened 3 weeks ago

paramazo commented 3 weeks ago

⚠️ This issue respects the following points: ⚠️

Bug description

Users reported that their folder view is sometimes empty, "No files found" gets displayed instead. After a few minutes or some retries it gets displayed. This happens after upgrade to NC v29.0.2.

Steps to reproduce

Click on files tab

Expected behavior

Files should be displayed

Installation method

Community Manual installation with Archive

Nextcloud Server version

29

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

None

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "29.0.2.2",
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "force_locale": "de_DE",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "proxy": "***REMOVED SENSITIVE VALUE***:8080",
        "proxyexclude": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "0": "log_level => 0",
        "onlyoffice": {
            "jwt_secret": "***REMOVED SENSITIVE VALUE***",
            "jwt_header": "AuthorizationJwt"
        },
        "maintenance": false,
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "pipe",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "has_rebuilt_cache": true,
        "versions_retention_obligation": "auto,3",
        "updater.release.channel": "stable",
        "loglevel": 2,
        "default_phone_region": "de",
        "mail_smtpsecure": "tls",
        "filelocking.enabled": true,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": { 
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "lost_password_link": "disabled",
        "maintenance_window_start": 1,
        "app_install_overwrite": [
            "files_mindmap"
        ]
    }
}

List of activated Apps

Enabled:
  - activity: 2.21.1
  - bruteforcesettings: 2.9.0
  - calendar: 4.7.6
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - comments: 1.19.0
  - contacts: 6.0.0
  - contactsinteraction: 1.10.0
  - dashboard: 7.9.0
  - dav: 1.30.1
  - deck: 1.13.0
  - drawio: 3.0.2
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_external: 1.21.0
  - files_lock: 29.0.1
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - flow_notifications: 1.9.0
  - forms: 4.2.4
  - groupfolders: 17.0.1
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - mail: 3.7.1
  - nextcloud_announcements: 1.18.0
  - notes: 4.10.0
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - onlyoffice: 9.2.2
  - password_policy: 1.19.0
  - photos: 2.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - spreed: 19.0.1
  - systemtags: 1.19.0
  - tables: 0.7.3
  - text: 3.10.0
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - updatenotification: 1.19.1
  - user_ldap: 1.20.0
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0

Nextcloud Signing status

No response

Nextcloud Logs

nextcloud.log is not logging anything, the web protocol reader hangs

Additional info

Not much load on the server

joshtrichards commented 3 weeks ago

After a few minutes or some retries it gets displayed.

Are you using External Storage? Are some your users' External Storage mounts inaccessible or slow?

nextcloud.log is not logging anything, the web protocol reader hangs

You can access the log file directory. It's located within your data/ directory.

Also, can you include the requested config report (output of occ config:list system or equivalent) and app list (occ app:list)?

paramazo commented 3 weeks ago

@joshtrichards

External Storage is not used.

The Logfile is located as you described, but without any new entries. Needs to set a loglevel in config?

Output of occ config:list system and occ app:list attached.

With the update to NC29 there was a new config maintenance_window_start which was set to 1. Now i set it to 100, the issue with missing files was not seen again yet.

nicnab commented 2 weeks ago

I'm not sure if this fits here but I have a similar problem on 29.0.3 for users on Safari 15.x (on macOS Catalina 10.15). Upon login, dashboard, files and deck are completely empty. Safaris on newer macOS versions work and so do all other browsers we use.

Specific Safari version for the following screenshots is 15.6.1 and macOS version 10.15.7:

Dashboard Deck Files

Same same, different existing bug or should I open a new bug? @paramazo do your users use an older version of Safari as well? Or maybe a different browser with an older version of Webkit?

paramazo commented 1 week ago

We are running occ files:scan every 30 minutes in a cron job. This prints sometimes updated and removed files. After running the cron we do not see this issue anymore.

nicnab commented 1 week ago

Interesting. Here the file scan does not make a difference. So it seems to be a different bug.

jasond2020 commented 1 week ago

The same issue - but in my case with activated external (samba) storage: Upgraded (from 27 step by step (first to 28 then) to NC 29.0.3.4 external storage folder empy (should be +300GB of files) -> running occ files:scan user and occ files:scan --all external storage folder show directories but most of them empty (only about 4GB of files) -> running occ files:scan --all again external storage folder shows about 10GB of files but mostly empty diretories -> running occ files:scan --all again and occ files:scan --unscanned --all no change

a second nextcloud in version 27.1.11.3 (recently updated to this version from 26) with the same configuration for external storage shows the complete content (all of the 300+ GB)

can this be related to the active/inactive memory cache? just something i noticed to be different in the two instances

paramazo commented 1 week ago

@jasond2020 We dont use external storage. For mem cache we have setup redis.

jasond2020 commented 1 week ago

After some more debugging and testing i can say, that the memcache active/inactive seemingly has nothing to do with this. But for sure the database error for the "occ_filecache" have. So maybe this is an individual problem OR (and that would be not good) this is an issue with NC 28 and/or 29 doing some bad stuff with the database while upgrading to the respective versions...

Hints in the Nextcloud-Forum and in other github issues lead me to this path (checking if everything is fine with e.g. innodb, utf encoding etc within the database) For the sake of documentation - at the moment i do not know how to repair the database (maybe somebody knows a way or maybe it is not possible at all):

MariaDB [nextcloud]> SHOW VARIABLES LIKE 'innodb_file_per_table'; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | innodb_file_per_table | ON | +-----------------------+-------+ 1 row in set (0,001 sec)

MariaDB [nextcloud]> ALTER DATABASE nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci; Query OK, 1 row affected (0,033 sec)

MariaDB [nextcloud]> CHECK TABLE oc_filecache;

+---------------------------+-------+----------+-----------------------------------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------------------------+-------+----------+-----------------------------------------------------------------------------------+ | nextcloud.oc_filecache | check | Warning | InnoDB: The B-tree of index fs_storage_mimetype is corrupted. | | nextcloud.oc_filecache | check | Warning | InnoDB: Index 'fs_storage_mimetype' contains 272968 entries, should be 611959. | | nextcloud.oc_filecache | check | Warning | InnoDB: The B-tree of index fs_storage_path_prefix is corrupted. | | nextcloud.oc_filecache | check | Warning | InnoDB: Index 'fs_storage_path_prefix' contains 611961 entries, should be 611959. | | nextcloud.oc_filecache | check | error | Corrupt | +---------------------------+-------+----------+-----------------------------------------------------------------------------------+ 5 rows in set (1 min 41,982 sec)

MariaDB [nextcloud]> REPAIR TABLE oc_filecache; +---------------------------+--------+----------+---------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------------------------+--------+----------+---------------------------------------------------------+ | nextcloud.oc_filecache | repair | note | The storage engine for the table doesn't support repair | +---------------------------+--------+----------+---------------------------------------------------------+ 1 row in set (0,092 sec)

occ maintenance:mimetype:update-db ... Added 79 new mimetypes Updated 39 filecache rows

MariaDB [nextcloud]> CHECK TABLE oc_filecache; +---------------------------+-------+----------+-------------------------------------------------------------+ | Table | Op | Msg_type | Msg_text | +---------------------------+-------+----------+-------------------------------------------------------------+ | nextcloud.oc_filecache | check | Warning | InnoDB: Index fs_storage_mimetype is marked as corrupted | | nextcloud.oc_filecache | check | Warning | InnoDB: Index fs_storage_path_prefix is marked as corrupted | | nextcloud.oc_filecache | check | error | Corrupt | +---------------------------+-------+----------+-------------------------------------------------------------+ 3 rows in set (51,348 sec)