`--define apc.enable_cli=1` flag in cron causes mysql db corruption

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

Steps to reproduce

Add the --define apc.enable_cli=1 flag to cron via crontab -u www -e according to the nextcloud documentation.
Open the Nextcloud installation in a browser of your choice (I'm using firefox).

Expected behaviour

The installation should run as intended.

Actual behaviour

Instead the following error is shown:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

The apache error log (see down below) leads towards the database. Trying to access it directly leads to this error:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (61)

Further testing revealed that the mysql service stopped and cannot be restarted:

root@nextcloud:~ # service mysql-server start
Starting mysql.
root@nextcloud:~ # service mysql-server status
mysql is not running.

The MariaDB error log:

Server version: 10.4.21-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467689 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
0x128342c <my_print_stacktrace+0x3c> at /usr/local/libexec/mysqld
0xc1c4f5 <handle_fatal_signal+0x295> at /usr/local/libexec/mysqld
0x8017c9b70 <_pthread_sigmask+0x530> at /lib/libthr.so.3
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: /var/db/system/cores/%N.core
2021-09-05 14:56:46 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-09-05 14:56:46 0 [Note] InnoDB: Uses event mutexes
2021-09-05 14:56:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-09-05 14:56:46 0 [Note] InnoDB: Number of pools: 1
2021-09-05 14:56:46 0 [Note] InnoDB: Using SSE2 crc32 instructions
2021-09-05 14:56:46 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-09-05 14:56:46 0 [Note] InnoDB: Completed initialization of buffer pool
2021-09-05 14:56:46 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1878312114
2021-09-05 14:56:46 0 [Note] InnoDB: Starting final batch to recover 31 pages from redo log.
2021-09-05 14:56:46 0x802761700  InnoDB: Assertion failure in file /wrkdirs/usr/ports/databases/mariadb104-server/work/mariadb-10.4.21/storage/innobase/log/log0recv.cc line 1669
InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
210905 14:56:46 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.4.21-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467689 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
0x128342c <my_print_stacktrace+0x3c> at /usr/local/libexec/mysqld
0xc1c4f5 <handle_fatal_signal+0x295> at /usr/local/libexec/mysqld
0x8017c9b70 <_pthread_sigmask+0x530> at /lib/libthr.so.3
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Core pattern: /var/db/system/cores/%N.core

Setting innodb_force_recovery (according to MariaDB docs) to level 6 out of 6, I was able to keep the service running. Attempting a table dump revealed in the logs:

2021-09-05 15:53:00 10 [ERROR] InnoDB: Table `nextcloud`.`oc_login_flow_v2` is corrupted. Please drop the table and recreate.
2021-09-05 15:56:37 13 [ERROR] InnoDB: Table `nextcloud`.`oc_login_flow_v2` is corrupted. Please drop the table and recreate.

I was able to recreate this issue the same way after resetting the database to a previous snapshot.

Server configuration

Operating system: TrueNAS Core 12.0-U5.1, running in a jail with release 12.2-RELEASE-p2.

Web server: Apache 2.4.48

Database: MariaDB 10.4.21

PHP version: PHP 8.0.10

Nextcloud version: 21.0.4

Updated from an older Nextcloud/ownCloud or fresh install: Updated from previous version, it seems like the since the cron job doesn‘t work any more since then.

Where did you install Nextcloud from: download.nextcloud.com

Signing status:

Signing status

``` No errors have been found. ```

List of activated apps:

App list

``` Accessibility 1.7.0 Activity 2.14.3 Brute-force settings 2.2.0 Contacts Interaction 1.2.0 Deleted files 1.11.0 End-to-End Encryption 1.7.1 External storage support 1.12.0 File sharing 1.13.1 First run wizard 2.10.0 Log Reader 2.6.0 Monitoring 1.11.0 Notifications 2.9.0 ONLYOFFICE 7.1.2 Password policy 1.11.0 PDF viewer 2.1.0 Photos 1.3.0 Privacy 1.5.0 Recommendations 1.0.0 Right click 1.0.0 Share by mail 1.11.0 Text 3.2.0 Theming 1.12.0 Two-Factor TOTP Provider 6.1.0 Update notification 1.11.0 Video player 1.10.0 Auditing / Logging 1.11.0 Collaborative tags 1.11.0 Comments 1.11.0 Dashboard 7.1.0 Default encryption module 2.9.0 Federation 1.11.0 LDAP user and group backend 1.11.0 Nextcloud announcements 1.10.0 Support 1.4.0 Usage survey 1.9.0 User status 1.1.1 Versions 1.14.0 Weather status 1.1.0 ```

Nextcloud configuration:

Config report

``` '***', 'passwordsalt' => '***', 'secret' => '***', 'trusted_domains' => array ( 0 => '10.0.0.12', 1 => 'cloud.***.de', ), 'trusted_proxies' => array ( 0 => '172.16.0.2', ), 'datadirectory' => '/mnt/data', 'dbtype' => 'mysql', 'version' => '21.0.4.1', 'overwrite.cli.url' => 'http://10.0.0.12', 'overwriteprotocol' => 'https', 'dbname' => 'nextcloud', 'dbhost' => 'localhost:/tmp/mysql.sock', 'dbport' => '', 'dbtableprefix' => 'oc_', 'mysql.utf8mb4' => true, 'dbuser' => '***', 'dbpassword' => '***', 'installed' => true, 'redis' => array ( 'host' => '/var/run/redis/redis.sock', 'port' => 0, ), 'memcache.local' => '\\OC\\Memcache\\APCu', 'memcache.locking' => '\\OC\\Memcache\\Redis', 'mail_smtpmode' => 'smtp', 'mail_sendmailmode' => 'smtp', 'mail_from_address' => 'admin', 'mail_domain' => '***', 'mail_smtpauthtype' => 'LOGIN', 'mail_smtpauth' => 1, 'mail_smtphost' => '10.0.0.11', 'mail_smtpport' => '1025', 'mail_smtpname' => '***', 'mail_smtppassword' => '***', 'default_phone_region' => 'DE', 'maintenance' => false, 'theme' => '', 'loglevel' => 2, ); ```

Are you using external storage, if yes which one: local storage, mounted to the jail

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Web server error log

``` [Sun Sep 05 16:39:05.753307 2021] [mpm_prefork:notice] [pid 16772] AH00163: Apache/2.4.48 (FreeBSD) configured -- resuming normal operations [Sun Sep 05 16:39:05.753697 2021] [core:notice] [pid 16772] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT' [Sun Sep 05 16:50:34.527561 2021] [proxy_fcgi:error] [pid 16777] [client 10.0.0.10:40961] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Doctrine\\DBAL\\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] Connection refused in /usr/local/www/nextcloud/lib/private/DB/Connection.php:85\nStack trace:\n#0 /usr/local/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1486): OC\\DB\\Connection->connect()\n#1 /usr/local/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1014): Doctrine\\DBAL\\Connection->getWrappedConnection()\n#2 /usr/local/www/nextcloud/lib/private/DB/Connection.php(226): Doctrine\\DBAL\\Connection->executeQuery()\n#3 /usr/local/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(210): OC\\DB\\Connection->executeQuery()\n#4 /usr/local/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(287): Doctrine\\DBAL\\Query\\QueryBuilder->execute()\n#5 /usr/local/www/nextcloud/lib/private/AppConfig.php(345): OC\\DB\\QueryBuilder\\QueryBuilder->execute()\n#6 /usr/local/www/nextcloud/lib/private/AppConfig.php(110): OC\\AppConfig->loadCo...' [Sun Sep 05 16:55:34.601321 2021] [proxy_fcgi:error] [pid 16774] [client 10.0.0.10:29494] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Doctrine\\DBAL\\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] Connection refused in /usr/local/www/nextcloud/lib/private/DB/Connection.php:85\nStack trace:\n#0 /usr/local/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1486): OC\\DB\\Connection->connect()\n#1 /usr/local/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1014): Doctrine\\DBAL\\Connection->getWrappedConnection()\n#2 /usr/local/www/nextcloud/lib/private/DB/Connection.php(226): Doctrine\\DBAL\\Connection->executeQuery()\n#3 /usr/local/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(210): OC\\DB\\Connection->executeQuery()\n#4 /usr/local/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(287): Doctrine\\DBAL\\Query\\QueryBuilder->execute()\n#5 /usr/local/www/nextcloud/lib/private/AppConfig.php(345): OC\\DB\\QueryBuilder\\QueryBuilder->execute()\n#6 /usr/local/www/nextcloud/lib/private/AppConfig.php(110): OC\\AppConfig->loadCo...' ```

Nextcloud log (data/nextcloud.log)

Nextcloud log

``` No current entries. ```

Browser log

``` No console statements. ```

nextcloud / server