jackblk / config

my config
0 stars 0 forks source link

Fix innodb incorrect file format corrupt in mariadb/mysql #29

Open jackblk opened 1 year ago

jackblk commented 1 year ago

Using https://hub.docker.com/r/bitnami/mariadb, I'm having this issue, can't start the container at all.

Checking logs with env var BITNAMI_DEBUG=1:

$ docker logs -f mariadb 
mariadb 11:43:34.29 
mariadb 11:43:34.30 Welcome to the Bitnami mariadb container
mariadb 11:43:34.30 Subscribe to project updates by watching https://github.com/bitnami/containers
mariadb 11:43:34.31 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mariadb 11:43:34.31 
mariadb 11:43:34.32 INFO  ==> ** Starting MariaDB setup **
mariadb 11:43:34.38 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 11:43:34.41 INFO  ==> Initializing mariadb database
mariadb 11:43:34.42 DEBUG ==> Ensuring expected directories/files exist
mariadb 11:43:34.44 INFO  ==> Updating 'my.cnf' with custom configuration
mariadb 11:43:34.45 INFO  ==> Setting user option
mariadb 11:43:34.46 DEBUG ==> Setting user to 'bn_wordpress' in mariadb configuration file /opt/bitnami/mariadb/conf/my.cnf
mariadb 11:43:34.49 INFO  ==> Setting slow_query_log option
mariadb 11:43:34.49 DEBUG ==> Setting slow_query_log to '0' in mariadb configuration file /opt/bitnami/mariadb/conf/my.cnf
mariadb 11:43:34.50 INFO  ==> Setting long_query_time option
mariadb 11:43:34.50 DEBUG ==> Setting long_query_time to '10.0' in mariadb configuration file /opt/bitnami/mariadb/conf/my.cnf
mariadb 11:43:34.51 INFO  ==> Using persisted data
mariadb 11:43:34.57 INFO  ==> Running mysql_upgrade
mariadb 11:43:34.58 INFO  ==> Starting mariadb in background
2022-12-25 11:43:34 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (server 10.9.4-MariaDB) starting as process 72 ...
2022-12-25 11:43:34 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-12-25 11:43:34 0 [Note] InnoDB: Using transactional memory
2022-12-25 11:43:34 0 [Note] InnoDB: Number of transaction pools: 1
2022-12-25 11:43:34 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-12-25 11:43:34 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2022-12-25 11:43:34 0 [Note] InnoDB: Using Linux native AIO
2022-12-25 11:43:34 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2022-12-25 11:43:34 0 [Note] InnoDB: Completed initialization of buffer pool
2022-12-25 11:43:34 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes)
2022-12-25 11:43:34 0 [Note] InnoDB: 128 rollback segments are active.
2022-12-25 11:43:34 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2022-12-25 11:43:34 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2022-12-25 11:43:34 0 [Note] InnoDB: log sequence number 47867740; transaction id 28273
2022-12-25 11:43:34 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2022-12-25 11:43:34 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-12-25 11:43:34 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-12-25 11:43:34 0 [Note] InnoDB: Buffer pool(s) load completed at 221225 11:43:34
2022-12-25 11:43:34 0 [Warning] 'proxies_priv' entry '@% root@a95d405e1aee' ignored in --skip-name-resolve mode.
2022-12-25 11:43:34 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.9.4-MariaDB'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution
Phase 1/7: Checking and upgrading mysql database
Processing databases
mysql
mysql.column_stats
Error    : Incorrect file format 'column_stats'
error    : Corrupt
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.global_priv                                  OK
mysql.gtid_slave_pos                               OK
mysql.help_category                                OK
mysql.help_keyword
Error    : Incorrect file format 'help_keyword'
error    : Corrupt
mysql.help_relation
Error    : Incorrect file format 'help_relation'
error    : Corrupt
mysql.help_topic
Error    : Incorrect file format 'help_topic'
error    : Corrupt
mysql.index_stats
Error    : Incorrect file format 'index_stats'
error    : Corrupt
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.roles_mapping                                OK
mysql.servers                                      OK
mysql.table_stats
Error    : Incorrect file format 'table_stats'
error    : Corrupt
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.transaction_registry                         OK

Repairing tables
mysql.column_stats
Error    : Incorrect file format 'column_stats'
error    : Corrupt
mysql.help_keyword
Error    : Incorrect file format 'help_keyword'
error    : Corrupt
mysql.help_relation
Error    : Incorrect file format 'help_relation'
error    : Corrupt
mysql.help_topic
Error    : Incorrect file format 'help_topic'
error    : Corrupt
mysql.index_stats
Error    : Incorrect file format 'index_stats'
error    : Corrupt
mysql.table_stats
Error    : Incorrect file format 'table_stats'
error    : Corrupt
Phase 2/7: Installing used storage engines... Skipped
Phase 3/7: Fixing views
mysql.user                                         OK
sys.host_summary                                   OK
sys.host_summary_by_file_io                        OK
sys.host_summary_by_file_io_type                   OK
sys.host_summary_by_stages                         OK
sys.host_summary_by_statement_latency              OK
sys.host_summary_by_statement_type                 OK
sys.innodb_buffer_stats_by_schema                  OK
sys.innodb_buffer_stats_by_table                   OK
sys.innodb_lock_waits                              OK
sys.io_by_thread_by_latency                        OK
sys.io_global_by_file_by_bytes                     OK
sys.io_global_by_file_by_latency                   OK
sys.io_global_by_wait_by_bytes                     OK
sys.io_global_by_wait_by_latency                   OK
sys.latest_file_io                                 OK
sys.memory_by_host_by_current_bytes                OK
sys.memory_by_thread_by_current_bytes              OK
sys.memory_by_user_by_current_bytes                OK
sys.memory_global_by_current_bytes                 OK
sys.memory_global_total                            OK
sys.metrics                                        OK
sys.processlist                                    OK
sys.ps_check_lost_instrumentation                  OK
sys.schema_auto_increment_columns                  OK
sys.schema_index_statistics                        OK
sys.schema_object_overview                         OK
sys.schema_redundant_indexes                       OK
sys.schema_table_lock_waits                        OK
sys.schema_table_statistics                        OK
sys.schema_table_statistics_with_buffer            OK
sys.schema_tables_with_full_table_scans            OK
sys.schema_unused_indexes                          OK
sys.session                                        OK
sys.session_ssl_status                             OK
sys.statement_analysis                             OK
sys.statements_with_errors_or_warnings             OK
sys.statements_with_full_table_scans               OK
sys.statements_with_runtimes_in_95th_percentile    OK
sys.statements_with_sorting                        OK
sys.statements_with_temp_tables                    OK
sys.user_summary                                   OK
sys.user_summary_by_file_io                        OK
sys.user_summary_by_file_io_type                   OK
sys.user_summary_by_stages                         OK
sys.user_summary_by_statement_latency              OK
sys.user_summary_by_statement_type                 OK
sys.version                                        OK
sys.wait_classes_global_by_avg_latency             OK
sys.wait_classes_global_by_latency                 OK
sys.waits_by_host_by_latency                       OK
sys.waits_by_user_by_latency                       OK
sys.waits_global_by_latency                        OK
sys.x$host_summary                                 OK
sys.x$host_summary_by_file_io                      OK
sys.x$host_summary_by_file_io_type                 OK
sys.x$host_summary_by_stages                       OK
sys.x$host_summary_by_statement_latency            OK
sys.x$host_summary_by_statement_type               OK
sys.x$innodb_buffer_stats_by_schema                OK
sys.x$innodb_buffer_stats_by_table                 OK
sys.x$innodb_lock_waits                            OK
sys.x$io_by_thread_by_latency                      OK
sys.x$io_global_by_file_by_bytes                   OK
sys.x$io_global_by_file_by_latency                 OK
sys.x$io_global_by_wait_by_bytes                   OK
sys.x$io_global_by_wait_by_latency                 OK
sys.x$latest_file_io                               OK
sys.x$memory_by_host_by_current_bytes              OK
sys.x$memory_by_thread_by_current_bytes            OK
sys.x$memory_by_user_by_current_bytes              OK
sys.x$memory_global_by_current_bytes               OK
sys.x$memory_global_total                          OK
sys.x$processlist                                  OK
sys.x$ps_digest_95th_percentile_by_avg_us          OK
sys.x$ps_digest_avg_latency_distribution           OK
sys.x$ps_schema_table_statistics_io                OK
sys.x$schema_flattened_keys                        OK
sys.x$schema_index_statistics                      OK
sys.x$schema_table_lock_waits                      OK
sys.x$schema_table_statistics                      OK
sys.x$schema_table_statistics_with_buffer          OK
sys.x$schema_tables_with_full_table_scans          OK
sys.x$session                                      OK
sys.x$statement_analysis                           OK
sys.x$statements_with_errors_or_warnings           OK
sys.x$statements_with_full_table_scans             OK
sys.x$statements_with_runtimes_in_95th_percentile  OK
sys.x$statements_with_sorting                      OK
sys.x$statements_with_temp_tables                  OK
sys.x$user_summary                                 OK
sys.x$user_summary_by_file_io                      OK
sys.x$user_summary_by_file_io_type                 OK
sys.x$user_summary_by_stages                       OK
sys.x$user_summary_by_statement_latency            OK
sys.x$user_summary_by_statement_type               OK
sys.x$wait_classes_global_by_avg_latency           OK
sys.x$wait_classes_global_by_latency               OK
sys.x$waits_by_host_by_latency                     OK
sys.x$waits_by_user_by_latency                     OK
sys.x$waits_global_by_latency                      OK
Phase 4/7: Running 'mysql_fix_privilege_tables'
ERROR 130 (HY000) at line 5: Incorrect file format 'column_stats'
ERROR 130 (HY000) at line 31: Incorrect file format 'help_topic'
ERROR 130 (HY000) at line 33: Incorrect file format 'help_relation'
ERROR 130 (HY000) at line 34: Incorrect file format 'help_keyword'
ERROR 130 (HY000) at line 35: Incorrect file format 'table_stats'
ERROR 130 (HY000) at line 36: Incorrect file format 'index_stats'
ERROR 130 (HY000) at line 58: Incorrect file format 'column_stats'
ERROR 130 (HY000) at line 77: Incorrect file format 'column_stats'
ERROR 130 (HY000) at line 146: Incorrect file format 'column_stats'
ERROR 130 (HY000) at line 164: Incorrect file format 'column_stats'
ERROR 130 (HY000) at line 237: Incorrect file format 'column_stats'
ERROR 130 (HY000) at line 299: Incorrect file format 'column_stats'
ERROR 130 (HY000) at line 366: Incorrect file format 'index_stats'
ERROR 130 (HY000) at line 505: Incorrect file format 'help_topic'
FATAL ERROR: Upgrade failed
mariadb 11:43:39.72 INFO  ==> Stopping mariadb
2022-12-25 11:43:39 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown
2022-12-25 11:43:39 0 [Note] InnoDB: FTS optimize thread exiting.
2022-12-25 11:43:39 0 [Note] InnoDB: Starting shutdown...
2022-12-25 11:43:39 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool
2022-12-25 11:43:39 0 [Note] InnoDB: Buffer pool(s) dump completed at 221225 11:43:39
2022-12-25 11:43:39 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2022-12-25 11:43:39 0 [Note] InnoDB: Shutdown completed; log sequence number 47867740; transaction id 28277
2022-12-25 11:43:39 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete

mariadb 11:43:40.73 DEBUG ==> Waiting for mariadb to unlock db files

To fix, firstly run a container with the same env vars that I made, but don't start the mysql/mariadb yet, it will fail because of the entrypoint script. Instead, create a bash shell so that we can work with mysql/mariadb.

docker run --rm -it -v "$(pwd)/data/mariadb:/bitnami/mariadb" \
 -e MARIADB_USER=bn_wordpress \
 -e MARIADB_DATABASE=bitnami_wordpress \
 -e MARIADB_PASSWORD=xxx \
 -e MARIADB_ROOT_PASSWORD=xxx \
 bitnami/mariadb:10.9 bash

Then, start mysql with a custom config, this will put innodb into recovery mode. Create a config file or append the current config file with this:

[mysqld]
innodb_force_recovery = 1

Note: for bitnami ijmage, the location of the binaries will be different, see reference: https://docs.bitnami.com/ibm/apps/wordpress/administration/recover-database-mariadb/

* Config: `/opt/bitnami/mariadb/my.cnf` * Start script: `/opt/bitnami/scripts/mariadb/run.sh`

To start mariadb in background: /opt/bitnami/scripts/mariadb/run.sh &

Now we can work with mariadb. Let's login: mysql -u root -p

After logging in successfully, repair the tables:

  MariaDB> use mysql;
  MariaDB> repair table user use_frm;
  MariaDB> check table user use_frm;;
  MariaDB> exit;

Everything should be good now.

Initially, I failed to repair the table because I missed use_frm. Refer this: http://wepeex.blogspot.com/2010/09/repair-mysql-tables-with-error.html (archive link)