Databases keep stopping on Mac with M1 and Pantheon recipes

andileco commented 3 years ago

Tell us about your setup

What is your lando version and operating system? (note that the older your version the less likely we are to reply)

Machine: MacBook Pro, BiG Sur v 11.5.2, Apple M1, 16GB memory Docker: 3.6.0 (packaged with Lando) Lando: 3.3.2

Tell us about your .lando.yml

name: my-site
recipe: pantheon
config:
  framework: drupal8
  site: my-site
  id: [the ID]

*pantheon.yml"

api_version: 1
web_docroot: true
php_version: 7.4
database:
  version: 10.4
enforce_https: full+subdomains
workflows:
  deploy_product:
    after:
      -
        type: webphp
        description: 'Run composer prepare-for-pantheon'
        script: private/scripts/deploy_product/prepare-for-pantheon.php
      -
        type: webphp
        description: 'Run composer install & drupal-scaffold'
        script: private/scripts/deploy_product/composer-install.php
  sync_code:
    after:
      - type: webphp
        description: Push changes back to GitHub if needed
        script: private/scripts/quicksilver/quicksilver-pushback/push-back-to-github.php

Tell us about the command you were running

After I run lando rebuild -y or lando start, the database basically immediately stops (I see this in the Docker dashboard), sometimes before even accessing the site, sometimes it will let me touch one or two pages before the container stops. I also stops in the middle of a lando pull - in this case, it generates an error like, "ERROR 2013 (HY000) at line 4753: Lost connection to MySQL server during query".

Tell us about the error you got

Other than the error during the database pull, there's no error, the database just stops. If I click into the database container, I see this:

Connection ID (thread ID): 8

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

Writing a core file...

Working directory at /bitnami/mariadb/data

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units     

Max cpu time              unlimited            unlimited            seconds   

Max file size             unlimited            unlimited            bytes     

Max data size             unlimited            unlimited            bytes     

Max stack size            8388608              unlimited            bytes     

Max core file size        0                    unlimited            bytes     

Max resident set          unlimited            unlimited            bytes     

Max processes             unlimited            unlimited            processes 

Max open files            1048576              1048576              files     

Max locked memory         65536                65536                bytes     

Max address space         unlimited            unlimited            bytes     

Max file locks            unlimited            unlimited            locks     

Max pending signals       10433                10433                signals   

Max msgqueue size         819200               819200               bytes     

Max nice priority         0                    0                    

Max realtime priority     0                    0                    

Max realtime timeout      unlimited            unlimited            us        

Core pattern: core

qemu: uncaught target signal 11 (Segmentation fault) - core dumped

/launch.sh: line 11:    39 Segmentation fault      /opt/bitnami/scripts/mariadb/entrypoint.sh /opt/bitnami/scripts/mariadb/run.sh

Tell us generally about your bug

This basically happens on all my Pantheon sites. SOMETIMES I can get the database to be stable for a few minutes by basically running a lot of Docker restarts, lando destroy, lando rebuild -y, lando start, but it doesn't seem like there's a perfect sequence of these commands that works every time.

Tell us more

Does this bug prevent you from using lando? YES

pirog commented 3 years ago

@andileco not surprising. we are still working through what things we need to change to reach full M1 support.

andileco commented 3 years ago

@pirog happy to be a test subject if you need someone to manually try fixes.

pirog commented 3 years ago

@andileco nice. definitely will take advantage of that.

we will be working on M1 support, specifically for the Pantheon recipe over the next two weeks!

cchoe1 commented 3 years ago

I'm currently on Pantheon and also running into troubles with the database. Attempting to run the exact same project on my Macbook Air M1, I can seemingly pull the database fine with lando pull but when I attempt to clear the site cache, I get this error:

If I attempt to just visit the site, I get this error:

It's the same errors everytime I try to re-pull and re-visit the site or re-clear the cache.

I've tried to rebuild the app. I've repulled multiple times thinking maybe it messed up somehow. I've tried with Docker 3.6 and 4.1 (I saw an issue where some people suggested there were improvements to mysql M1 compatibility in an rc2 branch that I believe has been committed as of the latest Docker version). More info here: https://github.com/docker/for-mac/issues/5389

Another thing I've noticed is that dropping tables is extremely slow. If I try to repull, it will first try to drop all tables and this goes at a snail's pace, maybe 1 small table every 10 seconds and some of the larger tables taking multiple minutes. Considering my site has over 900 tables, it can take quite a while just for the tables to get dropped. Running a lando destroy is far faster than waiting for tables to get dropped.

A third thing I've noticed, unrelated to databases, is that file syncing also takes forever. I ran a lando pull -d=none -f=live -c=none overnight and it wasn't finished by the time I woke up. In fact, it was in the middle of pulling these automated excel sheets on our site and it was at 4000 out of 32000 excel files, with each file being maybe like 50kb. Not a huge export by any means but it still took over 8 hours to not even be close to finishing. Not sure how many images it managed to pull before this which I imagine would take even longer.

Coufu commented 2 years ago

I'm running into the same problem. Any update on this one?

pirog commented 2 years ago

Something we hope to resolve in the next few weeks

Coufu commented 2 years ago

Thanks for the update @pirog ! Thank you and the team for all the hard work and appreciate the efforts towards supporting m1.

troseman commented 2 years ago

It might be that we need an arm64 database for m1. This is from docker's notes on apple silicon: "In particular, the mysql image is not available for ARM64. You can work around this issue by using a mariadb image. However, attempts to run Intel-based containers on Apple Silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation...In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only."

mariadb has arm64 version ready. tried this, https://hub.docker.com/r/arm64v8/mariadb/ but got an error with "/launch.sh: line 10: /entrypoint.sh: No such file or directory" in container. (but I barely know what I'm doing)

reynoldsalec commented 2 years ago

Alec's notes from chatting with Pirog about implementation here:

The Pantheon integration follows a slightly older pattern, so don't look for inspiration in the other integrations.

Normal integrations...

app.js is the entry point for app initialization
we do lots of corrections of core Lando behavior that isn't ideal
Biggest purpose: collect information that's specific to the integration, like collecting config files (pre-init)
Auto loading of services (usually descended from a parent type that does a lot of the work passing in environmental variables, other common procedures)

Panthon integration...

No services. Pantheon recipe just extends the LAMP recipe. Then configured in the builder.js with overrides.
Define database.version, should check if we can do more than 10.1/10.4...later version of D8/9 probably use 10.4

To do this task...

Create a new service (pantheon-database or pantheon-mariadb)
Model it (copy) after the existing mariadb service in lando-services/services/mariadb
Figure out how to get the Pantheon recipe to use this new DB service, possibly just by inserting the name of the new service in the recipe
Will need to either insert the new pantheon-mariadb option in loading the mysql tooling or copy over that tool loading to the Pantheon recipe builder.js
Switch the DB Service to use the core Docker Mariadb image, rewire env variables, DB usernames/passwords based on the Docker docs (the default of the LAMP recipe is to give the DB user/pass/dbname the same value as the recipe name)...we can hardcode those to "pantheon" to make sure upstream changes wouldn't screw with default operation.
Use the variable on the Mariadb container to allow an empty root user (so people can access that).
Make sure that the DB is volume mounted for persistent data (-v /var/lib/mysql) --> VERY IMPORTANT to make sure people don't lose data on rebuilds :) See the $(options.data):/bitnami/mariadb config in the mariadb builder.js, swap out for /var/lib/mysql.
Checkout out the pantheon-drupal7 Leia tests, update as necessary but these should be a good guide. We should add a test to check the version of the service and check the correct db name/username/password, and if they DB persists across rebuilds.

reynoldsalec commented 2 years ago

Merged; this will be in the next Lando release (3.6.0) which should be coming out tomorrow.

paulogabriel commented 2 years ago

I'm still getting this error after today's release (I'm on a arm64 mac) when I try to import database (db-import): Error response from daemon: Container 56daa31cc7ba12739dd81a21817a4c28429966d4f5a5b1cc682d7115e4bd4ca5 is not running Starting project_database_1 ... done

paulogabriel commented 2 years ago

also, when I run lando drush cr I get this:

In Connection.php line 190:
  SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo failed: Name or service not known                                                                  
In Connection.php line 190:
  PDO::__construct(): php_network_getaddresses: getaddrinfo failed: Name or service not known

labboy0276 commented 2 years ago

@paulogabriel did you do a destroy and start after you upgraded?

paulogabriel commented 2 years ago

@labboy0276 I did it all from scratch just to be sure... deleted the local repo, cloned it again, lando start for the first time then.

paulogabriel commented 2 years ago

@labboy0276 and yes, destroyed it as well before cloning/starting again

labboy0276 commented 2 years ago

@paulogabriel can you trying setting edge and cache to false in your lando file via:

recipe: pantheon
config:
  framework: PANTHEON_SITE_FRAMEWORK
  id: PANTHEON_SITE_ID
  site: PANTHEON_SITE_MACHINE_NAME
  edge: false
  cache: false

paulogabriel commented 2 years ago

@labboy0276 I get this:

Starting project_database_1        ... done
Starting project_index_1           ... done
Starting project_appserver_1       ... done
Starting project_appserver_nginx_1 ... done
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...
Waiting until database service is ready...

and keeps going

labboy0276 commented 2 years ago

@paulogabriel what do your logs say lando logs -s database

paulogabriel commented 2 years ago

@labboy0276

database_1         | userperms 01:03:43.42 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 01:03:43.64 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 01:03:43.67 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 01:03:43.69 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 01:09:50.73 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 01:09:50.86 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 01:09:50.89 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 01:09:50.91 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 01:21:06.88 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 01:21:06.98 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 01:21:07.00 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 01:21:07.01 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 02:07:30.78 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 02:07:30.88 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 02:07:30.90 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 02:07:30.91 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 02:33:13.25 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 02:33:13.45 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 02:33:13.49 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 02:33:13.51 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 02:34:26.86 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 02:34:27.08 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 02:34:27.14 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 02:34:27.22 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 02:35:05.11 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 02:35:05.29 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 02:35:05.34 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 02:35:05.37 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 16:12:23.48 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 16:12:23.62 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 16:12:23.65 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 16:12:23.67 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found

labboy0276 commented 2 years ago

@paulogabriel can you post the whole log from the beginning please

paulogabriel commented 2 years ago

@labboy0276 sure! here:

Attaching to project_database_1
database_1         | 
database_1         | 
database_1         | 
database_1         |                          STARTING UP                            
database_1         | 
database_1         |          ██       █████  ███    ██ ██████   ██████             
database_1         |          ██      ██   ██ ████   ██ ██   ██ ██    ██            
database_1         |          ██      ███████ ██ ██  ██ ██   ██ ██    ██            
database_1         |          ██      ██   ██ ██  ██ ██ ██   ██ ██    ██            
database_1         |          ███████ ██   ██ ██   ████ ██████   ██████             
database_1         | 
database_1         |        The best local development tool in the galaxy!                     
database_1         | 
database_1         | 
database_1         | ===============================================================
database_1         | 
database_1         | 
database_1         | userperms 01:03:43.42 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 01:03:43.64 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 01:03:43.67 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 01:03:43.69 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 01:09:50.73 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 01:09:50.86 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 01:09:50.89 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 01:09:50.91 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 01:21:06.88 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 01:21:06.98 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 01:21:07.00 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 01:21:07.01 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 02:07:30.78 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 02:07:30.88 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 02:07:30.90 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 02:07:30.91 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 02:33:13.25 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 02:33:13.45 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 02:33:13.49 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 02:33:13.51 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 02:34:26.86 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 02:34:27.08 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 02:34:27.14 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 02:34:27.22 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 02:35:05.11 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 02:35:05.29 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 02:35:05.34 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 02:35:05.37 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found
database_1         | userperms 16:12:23.48 WARN  ==> Only the root user can reset permissions! This is probably ok though...
database_1         | loadkeys 16:12:23.62 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
database_1         | lando 16:12:23.65 INFO  ==> Lando handing off to: docker-entrypoint.sh mysqld
database_1         | lando 16:12:23.67 DEBUG ==> Running command with exec...
database_1         | /lando-entrypoint.sh: 83: exec: docker-entrypoint.sh: not found

reynoldsalec commented 2 years ago

So far haven't been able to replicate this issue @paulogabriel on my M1; I suspect that this may be an issue specific to your installation, but I'll see if I can track down a mechanism for why this might occur.

paulogabriel commented 2 years ago

@reynoldsalec thank you for looking into this!

paulogabriel commented 2 years ago

@labboy0276 @reynoldsalec I just found out my problem was caused by a set of configuration in my lando.base file. I adjusted them and now database is being imported!!

reynoldsalec commented 2 years ago

@paulogabriel nice! Out of curiosity, what was in your config file that seemed to be the problem?

paulogabriel commented 2 years ago

@reynoldsalec

services:
   database:
      overrides:
         image: bitnami/mariadb:10.4.12

reynoldsalec commented 2 years ago

That makes total sense, thanks for clarifying @paulogabriel!

nkathiresshan commented 2 years ago

@paulogabriel what setting are you using now?

paulogabriel commented 2 years ago

@nkathiresshan I'm deleting that setting I commented above.

lando / pantheon

Databases keep stopping on Mac with M1 and Pantheon recipes #16