Closed dergachev closed 10 years ago
I have this same problem. And its getting really disruptive.
My config is as follows: ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-darwin12.4.0] gem -v -> 2.0.3 76 gems installed Vagrant 1.3.1 Berkshelf (2.0.10) vagrant-berkshelf (1.3.3) (the only vagrant plugin i am using) Test Kitchen version 1.0.0.beta.3
The test explained above (for N in {1..3} ; do time vagrant ; done) with and without the vagrant-berkshelf plugin goes from 0.33s to 1.85s. Six times slower!
I am using vagrant with berkshelf and test-kitchen (using serverspec) to migrate a group of physical servers to AWS. Step by step I have been creating the recipes necessary to provision all the servers' packages, middleware and proprietary software. This includes passing big tar.gz files to recreate some websites and to replicate databases.
Initially all went well and the TDD iterations were bearably fast. But as the recipes grew it became too slow.
An added problem is it is spending time doing things that are not needed. After a test has failed, I make a change and re-converge the VM previous to re-verify it: first berkshelf spends quite some time just resolving cookbook dependencies that have not changed and then it spends and awful lot of time copying cookbook resources that are already in the VM from the previous run. Only then chef begins working on the recipes. At this point resource idempotence works quite well.
The only way I have found to accelerate things a bit is commenting on the berksfile all the cookbooks that are already tested and the same thing in the .kitchen and vagrant files. But this is a pain in itself.
Now I was thinking about creating my own vagrant box with all the tested provisions on it after every successful iteration so I do not have to provision all every time. Far from an ideal workflow too.
Any help out there?
@mydocumenta let's not confound what this issue is about (vagrant-berkshelf needlessly requiring ALL its gems dependency every time I run something as simple as vagrant status
) and the fact that vagrant-berkshelf might be unnecessarily re-downloading unchanged cookbooks (I know it used to happen, but I'm not sure if that's the case anymore... hunt around the issue queue).
For your case, As a temporary workaround, consider using https://github.com/jimmycuadra/vagrant-librarian-chef or simply manually committing all your downloaded cookbooks to your git repo.
Also there are a few plugins that can really speed up iterating on this stuff:
... [berkshelf] spends and awful lot of time copying cookbook resources that are already in the VM from the previous run
This is done by test-kitchen, which unfortunately is using scp and not rsync. Berkshelf just copies them to a local directory which should be a fast operation (compared to all the other issues you listed).
Lazy loading the dependencies only when needed should indeed help some. But that should be done in both here and in Berkshelf itself.
Ok, thanks to both. I have been trying librarian-chef with the vagrant plugin and it is much faster (although it has its own problems).
You're right—vagrant-berkshelf makes Vagrant painfully slow. :/
However, I'd never go back to Librarian-Chef, which is IMO way less sophisticated. Instead, you could always use Berkshelf directly by running berks install --path cookbooks
before vagrant up/provision
. If you don't like having to run two commands instead of one, a simple Rake task might help.
@reset any thoughts on this?
For anyone else annoyed by this, I created a gem that works around this problem: https://github.com/dergachev/vgrnt
Note:
vgrnt
as a learning exercise, and don't expect to maintain itVAGRANT_NO_PLUGINS=1 vagrant ssh
is a far more robust alternative to vgrnt ssh
@dergachev yes, I have a few ideas to alleviate this problem.
@reseet thanks!
I am happy to see there's also progress on this here: https://github.com/berkshelf/berkshelf/issues/899
Ridley's host connector stuff has been moved out of Ridley core and Berkshelf is much faster. Additionally, Mitchell has done some awesome work in Vagrant to speed up plugins.
After installing vagrant-berkshelf, running
vagrant
executable has a 2+ second lag:Clearly the problem is
vagrant-berkshelf
, the fact that it requires a huge amount of dependent gems unconditionally.Here's one way to look at the bloat:
There are 75 (!!) gems that are downloaded as dependencies.
Here's a
Vagrantfile
that will help measure why it's slow:Let's install it and see what the culprits are. Additionally, we run a few diagnostic benchmarks to get a sense for the overhead of using bundler, etc:
The output from the invocation with PROFILE_REQUIRE=1 was:
From the above, it's obvious that
winrm
is a major part of the problem. In general though, we could shave 900ms ifvagrant-berkshelf
would only requireberkshelf
if it needs it. (However, my familiarity with berkshelf is insufficient to take a stab at this).Related issues: