Fix vagrant environment

[vkurup]

NOTE: This should be reviewed after #280 (Xenial upgrade) is merged because I did all my testing with those changes in place.

Issues addressed:

• Require upgrade of vagrant to version >= 1.8.7 because using a private networking address was fixed in that version. (I tested with the latest vagrant, which is 1.9.1)
• Because of the broken networking, we had switched to using 127.0.0.1:2222 in some places, but since it's working, I reverted those all to use 33.33.33.10 again
• NPM install caused OOM errors, so I had to bump the VM memory up to 2G
• Even after bumping up the memory, I was getting strange kernel errors which would cause the VM to be unreachable. If I was logged into the server at the time, all commands would result in an input/output error. Finally found this issue which refers to a VirtualBox problem causing VM corruption. Fortunately this has just been fixed in VirtualBox, but it requires you to uninstall the Ubuntu-provided VirtualBox and use the latest stable deploy from Oracle. I tested it, and it works. I added a check during the 'fab vagrant deploy' step to make sure that the vbox version was good before proceeding. (Couldn't figure out a way to do it cleanly in the 'vagrant up' step)
• Use a xenial box image
• The username on the xenial box image has changed from 'vagrant' to 'ubuntu'. Rather than hardcoding it again, I added code to get the proper username from vagrant and then to send that info via the pillar to margarita. See https://github.com/caktus/margarita/pull/148, which needs to be merged for this to work.

[mlavin]

NPM install caused OOM errors, so I had to bump the VM memory up to 2G

Um that seems problematic. Wouldn't we hit this on a VPC/EC2 instance with ~1G of memory then?

[vkurup]

I only tried once, but yes I hit an OOM error when trying to deploy to a t2.micro, so I've been using t2.small as my minimum for my Xenial tests.

[vkurup]

I also forgot to mention that the first time that I run 'fab vagrant setup_master', I always get this error:

vkurup@caktus020:~/dev/testdpt(master) $ fab vagrant setup_master
[localhost] local: vagrant ssh-config
[localhost] local: vboxmanage --version
[33.33.33.10] sudo: apt-get update -qq
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 427, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/home/vkurup/dev/testdpt/fabfile.py", line 158, in setup_master
    sudo('apt-get update -qq')
  File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 649, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 1109, in sudo
    stderr=stderr, timeout=timeout, shell_escape=shell_escape,
  File "/usr/local/lib/python2.7/dist-packages/fabric/operations.py", line 923, in _run_command
    channel=default_channel(), command=wrapped_command, pty=pty,
  File "/usr/local/lib/python2.7/dist-packages/fabric/state.py", line 397, in default_channel
    chan = _open_session()
  File "/usr/local/lib/python2.7/dist-packages/fabric/state.py", line 389, in _open_session
    return connections[env.host_string].get_transport().open_session()
  File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 159, in __getitem__
    self.connect(key)
  File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 151, in connect
    user, host, port, cache=self, seek_gateway=seek_gateway)
  File "/usr/local/lib/python2.7/dist-packages/fabric/network.py", line 575, in connect
    raise NetworkError(msg, e)
fabric.exceptions.NetworkError: Timed out trying to connect to 33.33.33.10 (tried 1 time)

If I run it immediately again, it works. I haven't figured that out yet, but would be interested to see if others see that problem as well.

[vkurup]

Thanks to some timely advice from @jbradberry in hipchat, I tried using a different vagrant box. Switching from the ubuntu-provided one to kaorimatz/ubuntu-16.04-amd64 fixes a bunch of things.

• It uses the vagrant user like we expect (and Vagrant recommends)
• It apparently doesn't use the same hard drive emulation because we don't hit that SCSI kernel bug, so we can use the default ubuntu VirtualBox package
• It includes 1 gigabyte of swap space, so we can keep the RAM used at 1G

The only minor gotcha was that it didn't have the apt-transport-https package installed by default, which we need for apt to be able to install HTTPS apt repos, so I added that to the accompanying PR over at caktus/margarita#148

You should be able to do:

vkurup@caktus020:~/dev/testdpt(master) $ vagrant up
vkurup@caktus020:~/dev/testdpt(master) $ time (fab vagrant setup_master && fab vagrant setup_minion:salt-master,db-master,cache,web,balancer -H 33.33.33.10 && fab vagrant deploy)
...
[33.33.33.10] out: Summary for ubuntu
[33.33.33.10] out: --------------
[33.33.33.10] out: Succeeded: 102 (changed=91)
[33.33.33.10] out: Failed:      0
[33.33.33.10] out: --------------
[33.33.33.10] out: Total states run:     102
[33.33.33.10] out: Total run time:   891.887 s
[33.33.33.10] out: 

Done.
Disconnecting from 33.33.33.10... done.

real    21m17.162s
user    0m17.392s
sys 0m5.176s

[jbradberry]

@vkurup the ssh timeout occurs with Ansible/Tequila as well, most of the time the first time a deploy is attempted on a fresh Vagrant box.

[vkurup]

@jbradberry Thanks for that info! Now that I know it's not just me, I added a retry around that first SSH connection. Now this command works for a 1-liner to set up a vagrant box:

vagrant destroy && vagrant up && fab vagrant setup_master && fab vagrant setup_minion:salt-master,db-master,cache,web,balancer -H 33.33.33.10 && fab vagrant deploy

[vkurup]

@dpoirier Updated the docs.

[dpoirier]

:steam_locomotive: