any more ideas for optimizing the Vlad build time?

[mvance]

I've been looking into ways to optimize the VLAD build time. So far, I've opened a couple of pull requests for enhancements that have helped me (one in ansible-role-solr and one in VLAD itself).

I just ran a couple builds from scratch with all the optional roles enabled. I then calculated a list of any tasks that took over one minute and came up with results below.

Anyone have other ideas that might help speed up any of these longer running tasks?

Personally, I'm especially interested in speeding up the install Ruby... and/or Install Drush dependencies... tasks, since those are longer running ones that I regularly use.

---

Task                                                        Run 1   Run 2
local actions | add custom identity from host to guest      1:33    1:36
ruby | install rbenv plugins                                1:16    1:09
ruby | install Ruby {{ default_ruby_version }}              7:19    7:11
drush | Install Drush dependencies with Composer            4:48    4:37
xhprof | install prerequisites for Debian                   1:31
xhprof | install xhprof                                             1:14
sendmail | Install Sendmail (Debian)                        2:26    2:08
solr | download Solr                                        3:35    6:32
ruby | install rbenv plugins                                1:07    1:25
mailcatcher | Install MailCatcher with rbenv                8:31    8:48

[dixhuit]

I'm especially interested in speeding up the install Ruby... and/or Install Drush dependencies... tasks, since those are longer running ones that I regularly use.

Ruby: Let's not worry too much about optimising a role that we're considering removing (#295).

Drush: Jeff Geerling's Drush role has evolved slightly since the current Vlad role was originally based on it. Since then I notice that it now includes the following which we may want to consider folding in to improve install performance?

drush_composer_cli_options: "--prefer-source --no-interaction" These options are the safest for avoiding GitHub API rate limits when installing Drush, and can be very helpful when working on dependencies/installation, but builds can be sped up substantially by changing the first option to --prefer-dist.

[mvance]

@danbohea

The "--prefer-source --no-interaction" options are already passed in within VLAD (though they aren't currently handled in a variable).

It looks like the Drush documentation is back to recommending a direct packaged download rather than a Composer-based build to install. Do you think it would make sense for VLAD and/or Jeff Geerling's Drush role to move in that direction? I think that a 4+ minute build process is enough of a reason to at least consider a direct packaged download.

[dixhuit]

The "--prefer-source --no-interaction" options are already passed in within VLAD (though they aren't currently handled in a variable).

Gaaagh! You're right. I forgot that it's the choice that Vlad's role is lacking, not the Composer options themselves.

It looks like the Drush documentation is back to recommending a direct packaged download rather than a Composer-based build to install.

So this looks like a recommendation for the global installation of Drush whereas Drupal 8 sites can still list a local version as a dep to be built via Composer if required.

Do you think it would make sense for VLAD and/or Jeff Geerling's Drush role to move in that direction? I think that a 4+ minute build process is enough of a reason to at least consider a direct packaged download.

I think that this is worth exploring and would be interested to see what time savings it introduces when provisioning. This may well be a good argument for abstracting Vlad's drush role to be a proper Galaxy role now.

We need to double check for any overlap between the final recommended installation step drush init and some of the tasks in the drush_extras role, in particular the handling of drushrc.php & .drush_bashrc. I'm not familiar with drush init, is that command available in all versions of drush that Vlad can install? I'm struggling to find the answer online, I guess we're gonna have to just do some tests if need be.

[dixhuit]

Update: I'm working on improving the drush role to support packaged downloads where possible. So far so good, I'll update this thread when the relevant code is pushed.

[dixhuit]

OK, there's now a new drush branch containing code that aims to add support for installing drush via a downloaded .phar file: https://github.com/hashbangcode/vlad/commit/66e8d053f7c198a4b7faff79f8a5d0553c19af92

This all ended up being more complex than first thought, hence the new branch. I'd really appreciate some further testing here if anyone's willing. So far I've only tested on Ubuntu 14 with various different values for drush_version (tagged releases & branch names).

In short though, you'll see a noticeable provisioning speed increase when setting drush_version to a tagged release >= 8.0.0. There's also a new variable drush_prefer_packaged_download that controls this preference which defaults to true. Also, please note that the default value of drush_version has now changed to 8.0.1.

From drush/defaults/main.yml:

---

# General
# ------------------------------------------------------------------------------

drush_install_path: /usr/local/share/drush
drush_path: /usr/local/bin/drush
drush_prefer_packaged_download: true

# Only applies to installation via Composer
drush_composer_path: /usr/local/bin/composer
drush_keep_updated: no

# Drush version
# ------------------------------------------------------------------------------

# Can be a release tag or a branch name.

# See Drush GitHub repo for options:
# - https://github.com/drush-ops/drush/releases
# - https://github.com/drush-ops/drush/branches

# Examples:
# - 7.x
# - master
# - 8.0.0

# NOTE: if drush_prefer_packaged_download is set to true, 
# drush_version will ideally need to be a release tag >= 8.0.0.

drush_version: 8.0.1

[zxaos]

As an aside here, it just occurred to me that an option for speeding up provisions past the first one would be to add vagrant-cachier to the required/auto-installed plugins list rather than the "use if you have it" list.

Or at least warning people that they should install it? Unless you read the Vagrantfile, it's not obvious the functionality is supported or you may not be aware the plugin exists...

[dixhuit]

I thought this was mentioned in the docs but you're right, it's not really clear enough. I'll at least get something sorted in the docs ASAP.

[dixhuit]

OK, I've now added mention of vagrant-cachier and other plugins to the Installation page of the docs (still just in dev for now). I'm sure this was covered at one stage but hey ho :)

[dixhuit]

Right, back to focussing on the drush branch. Please test and feedback! :)

[dixhuit]

I've done a bunch more testing on the drush branch now. There's a couple issues that I've raised today that make this a bit tricky when using an OS other than Ubuntu 14 (#314 & #315).

314 can be sidestepped by preventing the drupal_console role from running. I then tested Ubuntu 12 with PHP 5.3 & 5.4. The only issue here is if you try to install Drush 8 with PHP 5.3 - that's understandable as Drush 8 requires PHP >= 5.4, I'm happy to let this through but may revisit the code to possibly check the PHP version first and fail with a more helpful error if need be (though I'm keen to not rely on Vlad's php_version var for this as that will prevent this role from being abstracted to Ansible Galaxy).

315 makes testing on CentOS impossible right now and that's the only blocker to merging the drush role into dev that I can see. If you want to see this code in dev and then a tagged release, please take a look at #315 and help wherever you can :)

[dixhuit]

Now that #317 has been merged I have now successfully tested the changes to the drush role on CentOS (albeit 6.7 rather than 6.6). I'm going to merge the drush branch into dev shortly so that we can all benefit from the potential provisioning time improvement, especially while we test dev and it's new base boxes.

[dixhuit]

Drush role changes are now in dev, I've also updated the dev branch of the docs repo.

[mvance]

@danbohea,

Thanks for that drush install tweak!

Taking advantage of the newly added drush_prefer_packaged_download variable (as well as an optimized default_solr_source setting), I just tested with all the optional roles enabled and had only the following tasks take over a minute to run:

Task                                                        
local actions | add custom identity from host to guest      2:24
ruby | install rbenv plugins                                1:12
ruby | install Ruby {{ default_ruby_version }}              6:54
sendmail | Install Sendmail (Debian)                        2:16
solr | download Solr                                        2:20
ruby | install rbenv plugins                                1:10
mailcatcher | Install MailCatcher with rbenv                16:30

[dixhuit]

Good times :)

So 4 of those will be long gone when we ditch the ruby & mailcatcher roles for mailhog (hopefully we won't introduce any new slow tasks as a result).

That in effect leaves the following tasks with no planned optimisation at present:

Task                                                        
local actions | add custom identity from host to guest      2:24
sendmail | Install Sendmail (Debian)                        2:16
solr | download Solr                                        2:20

[dixhuit]

I find this result curious, especially as it's not reliant on internet connection speed:

local actions | add custom identity from host to guest      2:24

I generally have use_host_id set to true on projects but haven't noticed this task take as long as you've recorded (admittedly, I've never thought to time it). Maybe we should gather some more data here.

[dixhuit]

I've just noticed that the times in your first set of results aren't quite as severe.

[mvance]

I think the local actions | add custom identity from host to guest task includes the time it takes you to enter the password for the cross mounted directory, so that one is probably artificially long because I'm sometimes slow to notice or remember to enter the password.

[mvance]

Incidentally, in case anyone else is interested, I wrote a quick PHP script for tracking down long running tasks via the ansible.log file.