search

lando / mysql

The Official MySQL Lando Plugin
https://docs.lando.dev/mysql/
GNU General Public License v3.0
1 stars 2 forks source link

No Longer Able to start MySQL 5.7 Databases #6

Closed alextenczar closed 11 months ago

alextenczar commented 2 years ago

Hello,

My machine can no longer spin up MySQL containers for any new Lando Projects I create. Whenever I initiate "lando start" the everything goes smoothly until Lando attempts to start the MySQL container. At this point I continuously get the message "Waiting until database service is ready..." until it errors out and tells me that my database is unhealthy. Looking at the logs, the error message is:

"[ERROR] [FATAL] InnoDB: Aborting because of a corrupt database page in the system tablespace. Or, there was a failure in tagging the tablespace as corrupt.:

I've already tried destroying these projects and rebuilding them to no avail. This happens to every single project that was previously working. I uninstalled/reinstalled Docker, Docker-Compose, and Lando to no luck either.

Here is an example of a lando file that doesn't work:

name: lando-project recipe: drupal9 config: webroot: docroot php: '7.4' via: apache database: mysql:5.7 xdebug: false

I've been running Drupal projects exclusively, but it doesn't matter if it's 7, 8, or 9.

Note: If I choose Mariadb or MySQL 8.0 it works as usual. I would prefer not to be limited to only these though.

OS is Fedora 35 Lando: 3.6.0 Docker: 20.10.12 Docker-compose: 1.29.2

None of the documentation has been able to help me at this point. This has been driving me crazy.

Any Help would be greatly appreciated. Thank you.

alextenczar commented 2 years ago

Error from Lando after it's given up trying to launch the DB:

ERROR ==> message=, stack=Error at /snapshot/cli/lib/shell.js From previous event: at Shell.sh (/snapshot/cli/lib/shell.js) at Object.dc (/snapshot/cli/lib/bootstrap.js) at compose (/snapshot/cli/lib/bootstrap.js) From previous event: at /snapshot/cli/lib/router.js at processImmediate (internal/timers.js:464:21) From previous event: at Object.run (/snapshot/cli/lib/router.js) at run (/snapshot/cli/lib/engine.js) at /snapshot/cli/lib/router.js From previous event: at Object.eventWrapper (/snapshot/cli/lib/router.js) at Engine.engineCmd (/snapshot/cli/lib/engine.js) at Engine.run (/snapshot/cli/lib/engine.js) at /snapshot/cli/plugins/lando-core/app.js From previous event: at AsyncEvents. (/snapshot/cli/plugins/lando-core/app.js) at AsyncEvents.handle (/snapshot/cli/lib/events.js) at /snapshot/cli/lib/events.js at processImmediate (internal/timers.js:464:21) From previous event: at AsyncEvents.emit (/snapshot/cli/lib/events.js) at /snapshot/cli/lib/app.js From previous event: at App.start (/snapshot/cli/lib/app.js) at Object.run (/snapshot/cli/plugins/lando-core/tasks/start.js) at /snapshot/cli/lib/cli.js at processImmediate (internal/timers.js:464:21) From previous event: at /snapshot/cli/lib/cli.js From previous event: at Object.handler (/snapshot/cli/lib/cli.js) at Object.runCommand (/snapshot/cli/node_modules/yargs/lib/command.js:238:44) at Object.parseArgs [as _parseArgs] (/snapshot/cli/node_modules/yargs/yargs.js:1063:30) at Function.get [as argv] (/snapshot/cli/node_modules/yargs/yargs.js:1004:21) at Cli.init (/snapshot/cli/lib/cli.js) at Cli.run (/snapshot/cli/lib/cli.js) From previous event: at /snapshot/cli/bin/lando.js From previous event: at Object. (/snapshot/cli/bin/lando.js) at Module._compile (pkg/prelude/bootstrap.js:1751:22) at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10) at Module.load (internal/modules/cjs/loader.js:950:32) at Function.Module._load (internal/modules/cjs/loader.js:790:14) at Function.runMain (pkg/prelude/bootstrap.js:1804:12) at internal/main/run_main_module.js:17:47, stackCleaned=true Scanning to determine which services are ready... Please standby... Starting tuftshome_database_1 ... done ERROR ==> message=, stack=Error at /snapshot/cli/lib/shell.js From previous event: at Shell.sh (/snapshot/cli/lib/shell.js) at Object.dc (/snapshot/cli/lib/bootstrap.js) at compose (/snapshot/cli/lib/bootstrap.js) From previous event: at /snapshot/cli/lib/router.js at processImmediate (internal/timers.js:464:21) From previous event: at Object.run (/snapshot/cli/lib/router.js) at run (/snapshot/cli/lib/engine.js) at /snapshot/cli/lib/router.js From previous event: at Object.eventWrapper (/snapshot/cli/lib/router.js) at Engine.engineCmd (/snapshot/cli/lib/engine.js) at Engine.run (/snapshot/cli/lib/engine.js) at /snapshot/cli/plugins/lando-core/app.js From previous event: at AsyncEvents. (/snapshot/cli/plugins/lando-core/app.js) at AsyncEvents.handle (/snapshot/cli/lib/events.js) at /snapshot/cli/lib/events.js From previous event: at AsyncEvents.emit (/snapshot/cli/lib/events.js) at /snapshot/cli/lib/app.js From previous event: at App.start (/snapshot/cli/lib/app.js) at Object.run (/snapshot/cli/plugins/lando-core/tasks/start.js) at /snapshot/cli/lib/cli.js at processImmediate (internal/timers.js:464:21) From previous event: at /snapshot/cli/lib/cli.js From previous event: at Object.handler (/snapshot/cli/lib/cli.js) at Object.runCommand (/snapshot/cli/node_modules/yargs/lib/command.js:238:44) at Object.parseArgs [as _parseArgs] (/snapshot/cli/node_modules/yargs/yargs.js:1063:30) at Function.get [as argv] (/snapshot/cli/node_modules/yargs/yargs.js:1004:21) at Cli.init (/snapshot/cli/lib/cli.js) at Cli.run (/snapshot/cli/lib/cli.js) From previous event: at /snapshot/cli/bin/lando.js From previous event: at Object. (/snapshot/cli/bin/lando.js) at Module._compile (pkg/prelude/bootstrap.js:1751:22) at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10) at Module.load (internal/modules/cjs/loader.js:950:32) at Function.Module._load (internal/modules/cjs/loader.js:790:14) at Function.runMain (pkg/prelude/bootstrap.js:1804:12) at internal/main/run_main_module.js:17:47, stackCleaned=true

Here is the log:

database_1 | InnoDB: End of page dump database_1 | InnoDB: Page may be a freshly allocated page database_1 | 2022-02-11T01:52:09.252395Z 0 [ERROR] [FATAL] InnoDB: Aborting because of a corrupt database page in the system tablespace. Or, there was a failure in tagging the tablespace as corrupt. database_1 | 2022-02-11 01:52:09 0x7fef3b018740 InnoDB: Assertion failure in thread 140665463867200 in file ut0ut.cc line 918 database_1 | InnoDB: We intentionally generate a memory trap. database_1 | InnoDB: Submit a detailed bug report to http://bugs.mysql.com. database_1 | InnoDB: If you get repeated assertion failures or crashes, even database_1 | InnoDB: immediately after the mysqld startup, there may be database_1 | InnoDB: corruption in the InnoDB tablespace. Please refer to database_1 | InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html database_1 | InnoDB: about forcing recovery. database_1 | 01:52:09 UTC - mysqld got signal 6 ; database_1 | This could be because you hit a bug. It is also possible that this binary database_1 | or one of the libraries it was linked against is corrupt, improperly built, database_1 | or misconfigured. This error can also be caused by malfunctioning hardware. database_1 | Attempting to collect some information that could help diagnose the problem. database_1 | As this is a crash and something is definitely wrong, the information database_1 | collection process might fail. database_1 | database_1 | key_buffer_size=402653184 database_1 | read_buffer_size=2097152 database_1 | max_used_connections=0 database_1 | max_threads=151 database_1 | thread_count=0 database_1 | connection_count=0 database_1 | It is possible that mysqld could use up to database_1 | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1013732 K bytes of memory database_1 | Hope that's ok; if not, decrease some variables in the equation. database_1 | database_1 | Thread pointer: 0x0 database_1 | Attempting backtrace. You can use the following information to find out database_1 | where mysqld died. If you see no messages after this, something went database_1 | terribly wrong... database_1 | stack_bottom = 0 thread_stack 0x64000 database_1 | /opt/bitnami/mysql/bin/mysqld(my_print_stacktrace+0x2c)[0x55efcf4b698c] database_1 | /opt/bitnami/mysql/bin/mysqld(handle_fatal_signal+0x443)[0x55efcedd2963] database_1 | /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fef3b8e6730] database_1 | /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fef3b0547bb] database_1 | /lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fef3b03f535] database_1 | /opt/bitnami/mysql/bin/mysqld(+0x630d2f)[0x55efced99d2f] database_1 | /opt/bitnami/mysql/bin/mysqld(+0x631109)[0x55efced9a109] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z20buf_page_io_completeP10buf_page_tb+0x2e7)[0x55efcf6b8f77] database_1 | /opt/bitnami/mysql/bin/mysqld(+0xf7cd7e)[0x55efcf6e5d7e] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z13buf_read_pageRK9page_id_tRK11page_size_t+0x37)[0x55efcf6e64a7] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z16buf_page_get_genRK9page_id_tRK11page_size_tmP11buf_block_tmPKcmP5mtr_tb+0x471)[0x55efcf6bb341] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z24fsp_reserve_free_extentsPmmm13fsp_reserve_tP5mtr_tm+0xf8)[0x55efcf733328] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z19fseg_create_generalmmmmP5mtr_t+0x127)[0x55efcf739447] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z22trx_rseg_header_createmRK11page_size_tmmP5mtr_t+0x30)[0x55efcf657b90] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z15trx_rseg_createmm+0x1d8)[0x55efcf65afe8] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z20trx_sys_create_rsegsmmm+0x145)[0x55efcf65e4e5] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z34innobase_start_or_create_for_mysqlv+0x3822)[0x55efcf62e802] database_1 | /opt/bitnami/mysql/bin/mysqld(+0xd95710)[0x55efcf4fe710] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x55)[0x55efcee233f5] database_1 | /opt/bitnami/mysql/bin/mysqld(+0xb4e416)[0x55efcf2b7416] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x1dc)[0x55efcf2b92ec] database_1 | /opt/bitnami/mysql/bin/mysqld(+0x66295e)[0x55efcedcb95e] database_1 | /opt/bitnami/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x758)[0x55efcedcce08] database_1 | /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7fef3b04109b] database_1 | /opt/bitnami/mysql/bin/mysqld(_start+0x2a)[0x55efcedc377a] database_1 | The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains database_1 | information that should help you find out what is causing the crash.

labboy0276 commented 2 years ago

@alextenczar remove database key from your config section and try a custom image like:

services:
  database:
    type: mysql:custom
    overrides:
      image: 5.7.37-debian-10-r26

Does it work then?

alextenczar commented 2 years ago

@labboy0276 no luck. The image is unable to be found when I take that approach. If I specify the patch under config, the correct image installs but the database still fails.

Pulling database (5.7.37-debian-10-r26:)... ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image? [yN]y Pulling database (5.7.37-debian-10-r26:)... ERROR: pull access denied for 5.7.37-debian-10-r26, repository does not exist or may require 'docker login': denied: requested access to the resource is denied ERROR ==> message=, stack=Error at /snapshot/cli/lib/shell.js From previous event: at Shell.sh (/snapshot/cli/lib/shell.js) at Object.dc (/snapshot/cli/lib/bootstrap.js) at compose (/snapshot/cli/lib/bootstrap.js) at /snapshot/cli/lib/router.js at /snapshot/cli/lib/router.js at processImmediate (internal/timers.js:464:21) From previous event: at retryEach (/snapshot/cli/lib/router.js) at Object.start (/snapshot/cli/lib/router.js) at /snapshot/cli/lib/engine.js at /snapshot/cli/lib/router.js From previous event: at Object.eventWrapper (/snapshot/cli/lib/router.js) at Engine.engineCmd (/snapshot/cli/lib/engine.js) at Engine.start (/snapshot/cli/lib/engine.js) at /snapshot/cli/lib/app.js From previous event: at App.start (/snapshot/cli/lib/app.js) at Object.run (/snapshot/cli/plugins/lando-core/tasks/start.js) at /snapshot/cli/lib/cli.js at processImmediate (internal/timers.js:464:21) From previous event: at /snapshot/cli/lib/cli.js From previous event: at Object.handler (/snapshot/cli/lib/cli.js) at Object.runCommand (/snapshot/cli/node_modules/yargs/lib/command.js:238:44) at Object.parseArgs [as _parseArgs] (/snapshot/cli/node_modules/yargs/yargs.js:1063:30) at Function.get [as argv] (/snapshot/cli/node_modules/yargs/yargs.js:1004:21) at Cli.init (/snapshot/cli/lib/cli.js) at Cli.run (/snapshot/cli/lib/cli.js) From previous event: at /snapshot/cli/bin/lando.js From previous event: at Object. (/snapshot/cli/bin/lando.js) at Module._compile (pkg/prelude/bootstrap.js:1751:22) at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10) at Module.load (internal/modules/cjs/loader.js:950:32) at Function.Module._load (internal/modules/cjs/loader.js:790:14) at Function.runMain (pkg/prelude/bootstrap.js:1804:12) at internal/main/run_main_module.js:17:47, stackCleaned=true

reynoldsalec commented 11 months ago

@alextenczar LMK if you still have issues with the MySQL 5.7, I haven't had issues and looks like testing works ok, but never know what oddness exists...

alextenczar commented 11 months ago

Hi @reynoldsalec this was a very weird problem, but I eventually resolved it by switching my machine's distribution from Fedora to Ubuntu 22.04 LTS. I'm still not quite sure what happened, but I always assumed it may have been something with how Docker was set up. Anyways, sorted out on my end.