Provisioning with MariaDB taking nearly 20 minutes

stevegrunwell commented 2 years ago

It seems that version 11.4.0 of Settler has caused build times to increase dramatically when using MariaDB.

Since all of the supported PHP versions (5.6–8.0) now come pre-installed in the base box, Homestead wants to upgrade all of these versions, which means if the Settler box is at-all out of date the VM spends a lot of time updating anything out-of-date.

I've temporarily added a set +x to line 19 within scripts/features/mariadb.sh to get a better sense of where it all happens, which seems to be right as we're running apt-get remove -y --purge mysql-server mysql-client mysql-common (full output of the provisioning is linked below):

myapp: + apt-get remove -y --purge mysql-server mysql-client mysql-common
myapp: Reading package lists...
myapp: Building dependency tree...
myapp: Reading state information...
myapp: Package 'mysql-client' is not installed, so not removed
myapp: The following packages were automatically installed and are no longer required:
myapp:   gdal-data libaec0 libarmadillo9 libarpack2 libblas3
myapp:   libboost-serialization1.71.0 libcfitsio8 libcgi-fast-perl libcgi-pm-perl
myapp:   libcharls2 libdap25 libdapclient6v5 libencode-locale-perl libepsilon1
myapp:   libevent-core-2.1-7 libevent-pthreads-2.1-7 libfcgi-perl libfreexl1 libfyba0
myapp:   libgeos-3.8.0 libgeos-c1v5 libgeotiff5 libgfortran5 libgif7 libgmpxx4ldbl
myapp:   libhdf4-0-alt libhdf5-103 libhtml-parser-perl libhtml-tagset-perl
myapp:   libhtml-template-perl libhttp-date-perl libhttp-message-perl libio-html-perl
myapp:   libkmlbase1 libkmldom1 libkmlengine1 liblapack3 liblwp-mediatypes-perl
myapp:   libmecab2 libminizip1 libnetcdf15 libogdi4.1 libpoppler97 libproj15
myapp:   libprotobuf-c1 libqhull7 libsfcgal1 libsnmp-base libspatialite7 libsuperlu5
myapp:   libsz2 libtimedate-perl liburi-perl liburiparser1 libxerces-c3.2
myapp:   mecab-ipadic mecab-ipadic-utf8 mecab-utils mysql-client-core-8.0
myapp:   mysql-server-core-8.0 odbcinst odbcinst1debian2 proj-bin proj-data
myapp: Use 'sudo apt autoremove' to remove them.
myapp: The following additional packages will be installed:
myapp:   php5.6-bcmath php5.6-bz2 php5.6-cgi php5.6-cli php5.6-common php5.6-curl
myapp:   php5.6-dba php5.6-dev php5.6-enchant php5.6-fpm php5.6-gd php5.6-gmp
myapp:   php5.6-imap php5.6-intl php5.6-json php5.6-ldap php5.6-mbstring
myapp:   php5.6-mcrypt php5.6-mysql php5.6-odbc php5.6-opcache php5.6-pgsql
myapp:   php5.6-phpdbg php5.6-pspell php5.6-readline php5.6-recode php5.6-soap
myapp:   php5.6-sqlite3 php5.6-sybase php5.6-tidy php5.6-xml php5.6-xmlrpc php5.6-zip
myapp:   php7.0-bcmath php7.0-bz2 php7.0-cgi php7.0-cli php7.0-common php7.0-curl
myapp:   php7.0-dba php7.0-dev php7.0-enchant php7.0-fpm php7.0-gd php7.0-gmp
myapp:   php7.0-imap php7.0-intl php7.0-json php7.0-ldap php7.0-mbstring
myapp:   php7.0-mcrypt php7.0-mysql php7.0-odbc php7.0-opcache php7.0-pgsql
myapp:   php7.0-phpdbg php7.0-pspell php7.0-readline php7.0-recode php7.0-soap
myapp:   php7.0-sqlite3 php7.0-sybase php7.0-tidy php7.0-xml php7.0-xmlrpc php7.0-zip
myapp:   php7.1-bcmath php7.1-bz2 php7.1-cgi php7.1-cli php7.1-common php7.1-curl
myapp:   php7.1-dba php7.1-dev php7.1-enchant php7.1-fpm php7.1-gd php7.1-gmp
myapp:   php7.1-imap php7.1-intl php7.1-json php7.1-ldap php7.1-mbstring
myapp:   php7.1-mcrypt php7.1-mysql php7.1-odbc php7.1-opcache php7.1-pgsql
myapp:   php7.1-phpdbg php7.1-pspell php7.1-readline php7.1-recode php7.1-soap
myapp:   php7.1-sqlite3 php7.1-sybase php7.1-tidy php7.1-xml php7.1-xmlrpc php7.1-zip
myapp:   php7.2-bcmath php7.2-bz2 php7.2-cgi php7.2-cli php7.2-common php7.2-curl
myapp:   php7.2-dba php7.2-dev php7.2-enchant php7.2-fpm php7.2-gd php7.2-gmp
myapp:   php7.2-imap php7.2-intl php7.2-json php7.2-ldap php7.2-mbstring php7.2-mysql
myapp:   php7.2-odbc php7.2-opcache php7.2-pgsql php7.2-phpdbg php7.2-pspell
myapp:   php7.2-readline php7.2-recode php7.2-soap php7.2-sqlite3 php7.2-sybase
myapp:   php7.2-tidy php7.2-xml php7.2-xmlrpc php7.2-zip php7.3-bcmath php7.3-bz2
myapp:   php7.3-cgi php7.3-cli php7.3-common php7.3-curl php7.3-dba php7.3-dev
myapp:   php7.3-enchant php7.3-fpm php7.3-gd php7.3-gmp php7.3-imap php7.3-intl
myapp:   php7.3-json php7.3-ldap php7.3-mbstring php7.3-mysql php7.3-odbc
myapp:   php7.3-opcache php7.3-pgsql php7.3-phpdbg php7.3-pspell php7.3-readline
myapp:   php7.3-recode php7.3-soap php7.3-sqlite3 php7.3-sybase php7.3-tidy
myapp:   php7.3-xml php7.3-xmlrpc php7.3-zip php7.4-bcmath php7.4-bz2 php7.4-cgi
myapp:   php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-dev php7.4-enchant
myapp:   php7.4-fpm php7.4-gd php7.4-gmp php7.4-imap php7.4-intl php7.4-json
myapp:   php7.4-ldap php7.4-mbstring php7.4-mysql php7.4-odbc php7.4-opcache
myapp:   php7.4-pgsql php7.4-phpdbg php7.4-pspell php7.4-readline php7.4-soap
myapp:   php7.4-sqlite3 php7.4-sybase php7.4-tidy php7.4-xml php7.4-xmlrpc php7.4-zip
myapp:   php8.0-bcmath php8.0-bz2 php8.0-cgi php8.0-cli php8.0-common php8.0-curl
myapp:   php8.0-dba php8.0-dev php8.0-enchant php8.0-fpm php8.0-gd php8.0-gmp
myapp:   php8.0-imap php8.0-intl php8.0-ldap php8.0-mbstring php8.0-mysql php8.0-odbc
myapp:   php8.0-opcache php8.0-pgsql php8.0-phpdbg php8.0-pspell php8.0-readline
myapp:   php8.0-soap php8.0-sqlite3 php8.0-sybase php8.0-tidy php8.0-xml php8.0-zip
myapp: Suggested packages:
myapp:   dh-php
myapp: The following packages will be REMOVED:
myapp:   libgdal26* libmysqlclient21* libsnmp35* mysql-client-8.0* mysql-common*
myapp:   mysql-server* mysql-server-8.0* php5.6-snmp* php7.0-snmp* php7.1-snmp*
myapp:   php7.2-snmp* php7.3-snmp* php7.4-snmp* php8.0-snmp* postgresql-13-postgis-3*
myapp:   snmp*
myapp: The following packages will be upgraded:
myapp:   php5.6-bcmath php5.6-bz2 php5.6-cgi php5.6-cli php5.6-common php5.6-curl
myapp:   php5.6-dba php5.6-dev php5.6-enchant php5.6-fpm php5.6-gd php5.6-gmp
myapp:   php5.6-imap php5.6-intl php5.6-json php5.6-ldap php5.6-mbstring
myapp:   php5.6-mcrypt php5.6-mysql php5.6-odbc php5.6-opcache php5.6-pgsql
myapp:   php5.6-phpdbg php5.6-pspell php5.6-readline php5.6-recode php5.6-soap
myapp:   php5.6-sqlite3 php5.6-sybase php5.6-tidy php5.6-xml php5.6-xmlrpc php5.6-zip
myapp:   php7.0-bcmath php7.0-bz2 php7.0-cgi php7.0-cli php7.0-common php7.0-curl
myapp:   php7.0-dba php7.0-dev php7.0-enchant php7.0-fpm php7.0-gd php7.0-gmp
myapp:   php7.0-imap php7.0-intl php7.0-json php7.0-ldap php7.0-mbstring
myapp:   php7.0-mcrypt php7.0-mysql php7.0-odbc php7.0-opcache php7.0-pgsql
myapp:   php7.0-phpdbg php7.0-pspell php7.0-readline php7.0-recode php7.0-soap
myapp:   php7.0-sqlite3 php7.0-sybase php7.0-tidy php7.0-xml php7.0-xmlrpc php7.0-zip
myapp:   php7.1-bcmath php7.1-bz2 php7.1-cgi php7.1-cli php7.1-common php7.1-curl
myapp:   php7.1-dba php7.1-dev php7.1-enchant php7.1-fpm php7.1-gd php7.1-gmp
myapp:   php7.1-imap php7.1-intl php7.1-json php7.1-ldap php7.1-mbstring
myapp:   php7.1-mcrypt php7.1-mysql php7.1-odbc php7.1-opcache php7.1-pgsql
myapp:   php7.1-phpdbg php7.1-pspell php7.1-readline php7.1-recode php7.1-soap
myapp:   php7.1-sqlite3 php7.1-sybase php7.1-tidy php7.1-xml php7.1-xmlrpc php7.1-zip
myapp:   php7.2-bcmath php7.2-bz2 php7.2-cgi php7.2-cli php7.2-common php7.2-curl
myapp:   php7.2-dba php7.2-dev php7.2-enchant php7.2-fpm php7.2-gd php7.2-gmp
myapp:   php7.2-imap php7.2-intl php7.2-json php7.2-ldap php7.2-mbstring php7.2-mysql
myapp:   php7.2-odbc php7.2-opcache php7.2-pgsql php7.2-phpdbg php7.2-pspell
myapp:   php7.2-readline php7.2-recode php7.2-soap php7.2-sqlite3 php7.2-sybase
myapp:   php7.2-tidy php7.2-xml php7.2-xmlrpc php7.2-zip php7.3-bcmath php7.3-bz2
myapp:   php7.3-cgi php7.3-cli php7.3-common php7.3-curl php7.3-dba php7.3-dev
myapp:   php7.3-enchant php7.3-fpm php7.3-gd php7.3-gmp php7.3-imap php7.3-intl
myapp:   php7.3-json php7.3-ldap php7.3-mbstring php7.3-mysql php7.3-odbc
myapp:   php7.3-opcache php7.3-pgsql php7.3-phpdbg php7.3-pspell php7.3-readline
myapp:   php7.3-recode php7.3-soap php7.3-sqlite3 php7.3-sybase php7.3-tidy
myapp:   php7.3-xml php7.3-xmlrpc php7.3-zip php7.4-bcmath php7.4-bz2 php7.4-cgi
myapp:   php7.4-cli php7.4-common php7.4-curl php7.4-dba php7.4-dev php7.4-enchant
myapp:   php7.4-fpm php7.4-gd php7.4-gmp php7.4-imap php7.4-intl php7.4-json
myapp:   php7.4-ldap php7.4-mbstring php7.4-mysql php7.4-odbc php7.4-opcache
myapp:   php7.4-pgsql php7.4-phpdbg php7.4-pspell php7.4-readline php7.4-soap
myapp:   php7.4-sqlite3 php7.4-sybase php7.4-tidy php7.4-xml php7.4-xmlrpc php7.4-zip
myapp:   php8.0-bcmath php8.0-bz2 php8.0-cgi php8.0-cli php8.0-common php8.0-curl
myapp:   php8.0-dba php8.0-dev php8.0-enchant php8.0-fpm php8.0-gd php8.0-gmp
myapp:   php8.0-imap php8.0-intl php8.0-ldap php8.0-mbstring php8.0-mysql php8.0-odbc
myapp:   php8.0-opcache php8.0-pgsql php8.0-phpdbg php8.0-pspell php8.0-readline
myapp:   php8.0-soap php8.0-sqlite3 php8.0-sybase php8.0-tidy php8.0-xml php8.0-zip
myapp: 223 upgraded, 0 newly installed, 16 to remove and 112 not upgraded.
myapp: Need to get 59.2 MB of archives.

My first concern was that apt-get --purge remove <package> was implicitly running apt-get upgrade, but that would likely be big news in the Debian/Ubuntu community. Upon running it with the --no-upgrade flag on apt-get remove it still tried to upgrade everything.

That list of packages seems to match Settler's provisioning script, so the issue may be in that repository (which doesn't have an open issue tracker) — perhaps apt is holding onto a cache/staged set of upgrades?

Versions

Vagrant: Vagrant 2.2.18
Provider: VirtualBox 6.1.22
Homestead: 12.7.0 (Settler 11.4.0)

Host operating system

macOS 11.6

Homestead.yaml

ip: 192.168.10.42
memory: 2048
cpus: 2
provider: virtualbox
authorize: ~/.ssh/id_rsa.pub
keys:
  - ~/.ssh/id_rsa
folders:
  - map: /Users/steve/Code/myapp/app
    to: /home/vagrant/code
sites:
  - map: myapp.test
    to: /home/vagrant/code/public
    schedule: true
    php: '8.0'
databases:
    - myapp_development
services:
  - enabled:
    - mysql
    - redis-server
  - disabled:
    - beanstalkd
    - docker
    - memcached
features:
    - ohmyzsh: true
    - mariadb: true
    - postgres: false
name: myapp
hostname: myapp
in-flight-service: true

Vagrant destroy & up output

https://gist.github.com/stevegrunwell/1e878c5723d1b771f3d86fa322df5090

Expected behavior

I would expect the installation of MariaDB to add a moment or so to the provisioning of the Homestead box.

Actual behavior

Provisioning a Homestead box with MariaDB causes Ubuntu to try to install/upgrade all versions of PHP (5.6+) and their extensions, adding a considerable amount of time and bandwidth to the provisioning process. In all, provisioning Homestead takes nearly 20 minutes.

Steps to reproduce

Run vagrant up with the attached Homestead.yaml file, optionally prefixed with time (e.g. time vagrant up)

References

https://github.com/laravel/settler/releases/tag/v11.4.0

svpernova09 commented 2 years ago

~I want to ship homestead_mysql, homestead_mariadb base boxes.~ I want to ship homestead, homestead_mariadb base boxes I'm tired of dealing with MariaDB. I wish the base box building was more stable because at the moment it's very sketchy outside of Virtualbox.

svpernova09 commented 2 years ago

Can I get a 👍 to this comment if you're using MariadDB w/ Homestead?

svpernova09 commented 2 years ago

Homestead compatible base box with MariaDB out of the box

Matches Settler version v11.5.0

https://app.vagrantup.com/Svpernova09/boxes/nothomestead/versions/11.5.0

Build configuration:

SKIP_PHP=false
SKIP_MYSQL=true
SKIP_MARIADB=false
SKIP_POSTGRESQL=true

Override Homestead box

Update your Homestead.yaml adding the following:

box: Svpernova09/nothomestead

Adjusting your services:

services:
    - enabled:
        - "mariadb"

Note: Keep mariadb: false and mysql: true as feature: configuration for this testing session.

Vagrant up gist: https://gist.github.com/svpernova09/89468a18237b6c672bcdc0a9eede4fd0 (in 02:57 including Downloading the base box)

svpernova09 commented 2 years ago

Has anyone had a chance to test the above base box? Would appreciate some feedback.

thedavidthomas commented 2 years ago

@svpernova09 After running into issue #1717 due to me also updating Vagrant and Virtualbox at the same time I've installed the above base box successfully.

Two Runs: vagrant up 13.02s user 13.59s system 8% cpu 4:59.53 total (includes downloading the base box) vagrant up 6.45s user 4.04s system 4% cpu 3:47.34 total (importing the base box)

Most of the time spent seems to be installing php7.4 - See vagrant up gist when importing the base box: https://gist.github.com/thedavidthomas/35f3a0f598b7fcf821588deb75bcb892

Thank you for this under 5 mins is a great result

stevegrunwell commented 2 years ago

@svpernova09 My results were even better than what was reported by @thedavidthomas (after pre-downloading the base box):

vagrant up  13.80s user 9.95s system 15% cpu 2:29.44 total

Granted, this was a brand new, clean installation of Homestead without any site-specific configurations, but two and a half minutes! 🎉

Full results and steps: https://gist.github.com/stevegrunwell/30577c4dccd5d6436ea77fcd5c1c99c6

svpernova09 commented 2 years ago

@thedavidthomas @stevegrunwell

Thanks for the feedback! Glad this seems to help.

I'm currently uploading an Apple Silicon base box, after that I've had Taylor create a homestead-mariadb box as well. I'll get that uploaded next. (My upstream is awful). This base box will be compatible AND match the other settler versions (and such the laravel/homestead base box version) exactly with the swap of MariaDB instead of MySQL.

svpernova09 commented 2 years ago

I've released laravel/homestead-mariadb.

Override Homestead box

Update your Homestead.yaml adding the following:

box: laravel/homestead-mariadb

Adjusting your services:

services:
    - enabled:
        - "mariadb"

Note: Keep mariadb: false and mysql: true as feature: configuration.

laravel / homestead