Slower migrations and phpunit after moving to v7.9.0 and v7.10.0

[scoutman57]

Please note that the Homestead issue tracker is reserved for bug reports and enhancements. We are not always able to debug Vagrant, Provider or Operating System issues, but will do our best to help. Thank you!

Versions

• Virtualbox: 5.2.14 r123301
• Vagrant: 2.1.2
• Provider: Virtualbox
• Homestead: 7.6.2 | 7.9.0 | 7.10.0

A lot of issues can be resolved by simply updating vagrant, provider or homestead.

Note: Virtualbox users, please upgrade to ~5.2. You will show as up-to-date from the ~5.0 version when you About -> Check for Updates. You'll need to download a newer version of Virtualbox.

Host operating system

This is the operating system that you run locally.

Homestead.yaml

---
ip: "192.168.10.10"
memory: 2048
cpus: 2
provider: virtualbox
mariadb: true
ssl: true

authorize: ~/.ssh/id_rsa.pub

keys:
    - ~/.ssh/id_rsa

folders:
    - map: ~/Code
      to: /home/vagrant/Code

sites:
    - map: homestead.test
      to: /home/vagrant/Code/Laravel/public

databases:
    - testdb

# blackfire:
#     - id: foo
#       token: bar
#       client-id: foo
#       client-token: bar

# ports:
#     - send: 50000
#       to: 5000
#     - send: 7777
#       to: 777
#       protocol: udp

Vagrant destroy & up output

Provide a link to a GitHub Gist containing the complete output of vagrant destroy && vagrant up. Do NOT paste the debug output in the issue, just paste the link to the Gist.

Expected behavior

I would expect that after upgrading to unbuntu 18.04 (homestead 7.10.0) that the migration and phpunit run times would not be significantly slower in the newer version. If anything one would expect the run times to be faster in the newer versions.

Actual behavior

The migration and phpunit run times dramatically increased after upgrading to the newer version of homestead.

Steps to reproduce

1. create a decent about of migrations (I have about 25)
2. run migrations in homestead v7.6.2 "time php artisan migrate" (note times) repeat for PHPUnit
3. upgrade homestead to v7.10.0
4. run migrations in homestead v7.10.0 "time php artisan migrate" (note times) repeat for PHPUnit

My migration times were 2.647s on homestead v7.6.2 they went up to 17.747s (these times are greatly different with the exact same migrations, the only difference was the homestead version). The times for phpunit test to execute also dramatically increased which I believe is because of the increased times of the migrations. Most of my unit test took 4 seconds to run on 7.6.2 and now take 20-30s on 7.10.0. This issue appears to have introduced after moving to Ubuntu 18.04

I could reproduce this one two different system (MacBook Pro & Hackintosh) Both running High Serria.

I did not test this in Homestead v7.7.0 or v7.8.0 but noticed this a couple of weeks ago in 7.9.0 but just now was able to get around to testing. Before 7.9.0 I was on 7.6.2.

[scoutman57]

The times are dramatically different in 7.7.0 as well. Since this is the first time Unbutu 18.04 was introduced, I am not testing v7.8.0.

Also, I tested MySQL and the times are still dramatically different still so this issue does not seem to be an issue related to specifically MariaDB or MySQL. MySQL on v7.6.2 ran migrations at 4.629s and on v7.10.0 the migrations ran at 28.178s

[browner12]

Have you used the same Homestead.yaml throughout all tests? so the amount of memory and number of CPUs has been consistent?

[svpernova09]

Update your Homestead.yaml to with these changes and test again to see if you still see the same results:

folders:
    - map: ~/Code
      to: /home/vagrant/Code/Laravel

sites:
    - map: homestead.test
      to: /home/vagrant/Laravel/public

[scoutman57]

@svpernova09 I have actually already tried that and it makes zero difference. I was thinking there was a "too many files" issue but that's not the case as this is the only code base in the mapped folders. I actually tried all this before going into full depth testing. The times don't change.

[scoutman57]

@browner12 Yes, the Homestead.yaml did not change. That is the only way to make a valid comparison. The only thing that changed was the homestead git branches, hence the base box. To get 7.6.2 to use the older vagrant box you have to change the config.vm.box_version in homestead.rb from config.vm.box_version = settings["version"] ||= ">= 5.2.0" to config.vm.box_version = settings["version"] ||= "5.2.0" or else even v7.6.2 will try to download the latest vagrant box which is 6.0.0

[svpernova09]

FYI You shouldn't override the config.vm.box_version that's the only way we have to keep you from using the wrong box. You should always use the current tagged version of the repo and the base box since those are the only versions we'll support here.

I wouldn't be able to debug this any farther without seeing your application's source.

[scoutman57]

I already know we should NOT override the config.vm.box_version but that is the only way to make vagrant pull the older version of the homestead box for test. I know why it's done the way it's done.

I am not sure my application's source is the problem as I have now tested this on 3 different Laravel projects. All of them is vastly different and all of them running migration and phpunit slower on the 18.04 box than on the 16.04 box with the exact same Homestead.yaml setting for both vagrant boxes and homestead mapped to the exact same project directories for consistency.

[svpernova09]

What are the specs of your host machine and what operating system and what version?

[svpernova09]

Are any of the projects publicly available so we can test?

[scoutman57]

getting specs. but I believe the unit test is slow because of the "use RefreshDatabase;" in all the unit test files. So to take them out of the equation I moved to just test migrations on both boxes using "time php artisan migrate:fresh"

[scoutman57]

I am going to download a public laravel project and test with that.

[scoutman57]

My specs for hackintosh: macOS High Sierra Processor: 4.2 GHz Intel Core i7 Memory: 64GB 2133 MHz DDR4 HHD: NVMe M.2 960 EVO

MacBook Pro macOS High Sierra Processor: 2.3 GHz Intel Core i7 Memory: 16 GB 1600 MHz DDR3 HHD: SSD (Whatever the late 2013 model has)

[svpernova09]

I can't speak to the hackintosh, but it wouldn't surprise me to see the Macbook Pro take a little longer. (still odd that it shot up to 17.x seconds)

Here is the repo that I use to test Homestead: https://github.com/svpernova09/quickstart-5.5 (Checkout brand exercise-9, it's not heavy on migrations however).

Unless we can get someone to reproduce, there's not much we can do to help here.

[scoutman57]

I know the Macbook would be slower but it should not be significantly slower depending on the version of homestead. I downloaded a fresh copy of laravel and benched it in both and they were consistently slower in 18.06 by 2s (does not seem like a big difference, but that is just two migrations. After lots more research and troubleshooting/testing with another developer, we came across this issue in docker. https://github.com/docker/for-linux/issues/247

We changed these setting from "8" to "0" in both locations

echo 0 >/sys/block/sda/queue/iosched/slice_idle
echo 0 >/sys/block/sda/queue/iosched/group_idle

This change dramatically sped up the migrations in 18.04. I went back after the change and tested my previous projects. These projects now run consistently around the same amount of time as 16.04.

These settings are for older legacy spinning drives and not SSD drives but it still utilizes the settings. SEE SPEC https://www.kernel.org/doc/Documentation/block/cfq-iosched.txt

I have changed the settings from 8 to 0 and back several times and the migration times are consistent with 16.04 when set to "0" and greatly slower when set to "8"

[scoutman57]

Just for some ideas of my IO disk speeds on the Hackintosh SSD. https://lh3.googleusercontent.com/M88Yk3Qok8y6xPNFksFUV9-0CZ1TRUQLJ1UMfWtTIPhn5WtNpAB-IIqiyRunDZ_3fX6xV4KmYE_a1mKhWsaHQv7aeenD74-qquZxRk2LNzSugExSP7H4TAX-cJjOVg2AZcv9HikTsXY=w530-h278-p-rw

[svpernova09]

Wow, interesting.thanks for sharing

[scoutman57]

No problem, it was a pain in the %!@#%SDf to debug. We tested everything from UUID create to different speeds in unique vs index, speeds without indexes ... fyi unique is a little slower than index. Changing the values took my times dramatically lower in my application.

Here is a good test for getting some average times to execute 10 iterations of your migrations: time seq 10 | xargs -Iz php artisan migrate:fresh --quiet

18.04 went from 2m44.637 to 0m34.887 after changing the values to "0".

I hope that maybe this is something you can build into the vagrant box, or a setting we can define in the Homestead.yaml. Something like SSD: true and then vagrant code can change those settings to "0" through the magic of ruby!

[svpernova09]

https://github.com/laravel/homestead/releases/tag/v7.12.0 Adds the disable_cfq option.

Usage in Homestead.yaml:

disable_cfq: true

[sebastiaanluca]

This seems to cut my migrations' run time in half (from 3 minutes to 1.5), but the provisioning command to set those values doesn't seem to work? New Homestead VM and values stay at 8 after boot and provision.

Doesn't work (https://github.com/laravel/homestead/pull/898/files):

sudo echo 0 >/sys/block/sda/queue/iosched/slice_idle
sudo echo 0 >/sys/block/sda/queue/iosched/slice_idle

Works:

sudo sh -c "echo '0' >> /sys/block/sda/queue/iosched/slice_idle"
sudo sh -c "echo '0' >> /sys/block/sda/queue/iosched/group_idle"

Edit: this change is also temporary, so it currently resets to 8 on reboot. We'd have to set up some script to apply changes on boot. For reference: https://github.com/fossology/fossology/wiki/Improving-IO-performance.

No luck with the general NFS read/write slowness of the VM though (https://github.com/laravel/homestead/issues/901). Takes half a minute to load a simple page :(

[svpernova09]

@sebastiaanluca sounds like we need to update the commands on the disable_cfq: true?

No luck with the general NFS read/write slowness of the VM though (#901). Takes half a minute to load a simple page :(

NFS....it's a beast with virtualization :/

[sebastiaanluca]

Update the commands and unless Homestead.rb runs on every boot, they'd have to run in a boot script so they're not applied just for the current session.

NFS....it's a beast with virtualization :/

True, but haven't found the real cause yet. A default Ubuntu box provisioned with Ansible and using NFS for synced folders runs fine and fast. Homestead by default takes 22+ seconds to load a single page and 3 minutes to run our migrations. With this fix applied, I can cut that down to 8 seconds and 1.5 minutes, but it's still quite unusable. So this leads me to think it's a setting somewhere that needs tweaking, but who knows …

Currently thinking of provisioning a box with Ansible and using Homestead to create sites and sync folders. Maybe that'll work better ¯_(ツ)_/¯

[svpernova09]

Can you share the ansible playbook and / or the Vagrantfile + configuration so I can compare w/ Homestead?

[scoutman57]

I have opened a pull request that should resolve the value not being set properly

On Fri, Feb 8, 2019 at 11:06 AM Sebastiaan Luca notifications@github.com wrote:

Update the commands and unless Homestead.rb runs on every boot, they'd have to run in a boot script so they're not applied just for the current session.

NFS....it's a beast with virtualization :/

True, but haven't found the real cause yet. A default Ubuntu box provisioned with Ansible and using NFS for synced folders runs fine and fast. Homestead by default takes 22+ seconds to load a single page and 3 minutes to run our migrations. With this fix applied, I can cut that down to 8 seconds and 1.5 minutes, but it's still quite unusable. So this leads me to think it's a setting somewhere that needs tweaking, but who knows …

Currently thinking of provisioning a box with Ansible and using Homestead to create sites and sync folders. Maybe that'll work better ¯(ツ)/¯

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/laravel/homestead/issues/896#issuecomment-461873857, or mute the thread https://github.com/notifications/unsubscribe-auth/AA2C1UkBrr0mJqxYCrF6qaclvrcq3Adtks5vLa6ogaJpZM4VOV7X .

[scoutman57]

Joe, I was about to ask the same things. I am wondering if it's puppet related. I was thinking of trying to do the same thing with SALT or Chef

On Fri, Feb 8, 2019 at 11:17 AM Joe Ferguson notifications@github.com wrote:

Can you share the ansible playbook and / or the Vagrantfile + configuration so I can compare w/ Homestead?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/laravel/homestead/issues/896#issuecomment-461877462, or mute the thread https://github.com/notifications/unsubscribe-auth/AA2C1YuGGS3PYLClJMIQN5OGM48__O1Aks5vLbEugaJpZM4VOV7X .

[scoutman57]

Not puppet lol

On Fri, Feb 8, 2019 at 11:20 AM Shannon Warren scoutman57@gmail.com wrote:

Joe, I was about to ask the same things. I am wondering if it's puppet related. I was thinking of trying to do the same thing with SALT or Chef

On Fri, Feb 8, 2019 at 11:17 AM Joe Ferguson notifications@github.com wrote:

Can you share the ansible playbook and / or the Vagrantfile + configuration so I can compare w/ Homestead?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/laravel/homestead/issues/896#issuecomment-461877462, or mute the thread https://github.com/notifications/unsubscribe-auth/AA2C1YuGGS3PYLClJMIQN5OGM48__O1Aks5vLbEugaJpZM4VOV7X .

[sebastiaanluca]

Can you share the ansible playbook and / or the Vagrantfile + configuration so I can compare w/ Homestead?

Don't have anything ready to share right now. Will have to set up an example project first to illustrate the speed difference which can be tested by a few. Maybe it depends on which machine it runs on (even with the same amount of CPUs and memory).

But in short: basically just the most default Vagrantfile using the ubuntu/bionic64 box, 8GB of memory, and 4 CPUs (same variables as with Homestead) provisioned with PHP, nginx, and MySQL using geerlingguy's packages. I have a ton more stuff added like Redis, etc but the basics should suffice. If it works fast with those extras, it should work fast(er) without.