Avoid installing (i.e. compiling) ruby from source during vagrant up?

[DeeDeeG]

Hi all,

One of the biggest time sinks right now when running vagrant up is compiling ruby from source. I wonder if we could compile once, host the compiled version in, say https://github.com/refugerestrooms/third-party-ruby or something like that, then download and install the binary from there, instead of compiling every time we make a new vm?

Right now vagrant up runs somehwere near and hour to an an hour and a half for me. I think it would be super convenient and great if we could bring that down to a more manageable, quicker provision. More convenient testing = more testing and happier devs?

[DeeDeeG]

Research links

Easy tutorial

How to make a package as a .deb https://ubuntuforums.org/showthread.php?t=910717

Detailed tutorials

More detailed version of how to make a package as a .deb (intended for those that want their package accepted into ubuntu or debian package repositiories, which we don't need to do. Good advice, regardless.) http://packaging.ubuntu.com/html/packaging-new-software.html

Alternative detailed version (step-by-step instructions, where the author packages a real piece of software, explains what goes wrong and how to fix it. More of a Debian than Ubuntu tutorial, but would work fine with optional tweaks for a Ubuntu package) https://wiki.debian.org/Courses2005/BuildingWithoutHelper

[mi-wood]

Another way to do this might be creating our own vagrant base box that already has ruby and the other dependencies set up. We could host this somewhere. Maybe there's something free, or s3 will cost practically nothing for this. Then people just have to download that box and it's all set up. The setup script is idempotent, so it can always be run for minor updates after that.

[DeeDeeG]

Idempotent

I learned a new vocab word today!

That all sounds good to me. Just popping in again to say that Nokogiri is the other large-ish consumer of time during vagrant up, as it too compiles from source. Although I see on its website that is by design.

Another way to do this might be creating our own vagrant base box

Wow, that would be amazing.

I found a link to a lengthy but relatively easy-looking tutorial for creating the box: https://github.com/hollodotme/Helpers/blob/master/Tutorials/vagrant/self-hosted-vagrant-boxes-with-versioning.md

I assume this isn't strictly the only way to do it, but it answers most of the unknowns from my perspective.

Also, if downloading a box from some third party seems sketchy (it does to me), this tutorial does the first steps starting from scratch, with Ubuntu downloaded from Ubuntu.com: https://www.sitepoint.com/create-share-vagrant-base-box/

[DeeDeeG]

Edit: Unless someone says otherwise, I will lean toward doing this the vagrant box way first.

If we would be comfortable moving away from rbenv, I would be interested in moving forward with the .deb packaging of ruby. That seems less daunting to me than the vagrant box solution.

The Ubuntu bug tracker thingy, "Launchpad.net," has a .deb package hosting solution, for free, baked into it, so we wouldn't have to worry about hosting costs/logistics.

For integrating it with our vagrant up setup process: I could upload the .deb at launchpad, as mentioned above. We could then have the script download the .deb from launchpad.net, with wget, then install it with apt install ./[package_name].deb

Regarding the vagrant box route: there are a couple of kind-of-hacky ways to do this, other than paying Amazon to host the file. One of us could set up a home server and serve the files from there (probably a bad idea). We could throw it on a dropbox account or similar and download from there (maybe against terms of service, would have to double check). Kind of sneaky, and not sure if against the terms of service, we could host the boxes on GitHub, if possible (may run into some sort of filesize limit). (Looking at just the terms of service, I think we are fine.)

Edit to add, regarding S3 pricing: S3 pricing is really cheap! https://aws.amazon.com/s3/pricing/ Editing again: hosting vagrant boxes on GitHub for free is a no-go, max filesize is 100MB. A typical box is roughly 400-800 MB or so?

[DeeDeeG]

I've been working on this. I made the box (it's much faster to run vagrant up starting with it), but so far I can't think of a way to host it for free and still have it automatically download during vagrant up.

Here's a link to grab the box file I made: https://www.dropbox.com/s/pc7wi24swy5iql2/refugerestrooms-box-test-1.0.box?dl=0

(Alternate download for the box. This is smaller \~500 MB vs \~850 MB. https://drive.google.com/open?id=0B3gIqnvjGoJsMTFLVEoza2FrM0U)

New link November 2017 (Rails 5.1.4 and updated gems, ~440MB): https://drive.google.com/open?id=0B3gIqnvjGoJsMWNJTVd3RUdCQ2s

You can use it like this:

• Download the .box file from one of the links above.
• Change line 10 in the Vagrantfile from: config.vm.box = "ubuntu/trusty64" to: config.vm.box = "/home/[username]/Downloads/[filename.box]"

• vagrant up and it will run as usual.

• (Optional): To preserve an existing refugerestrooms virtualbox machine, edit your Vagrantfile to have BOXNAME = "refugerestrooms-box-testing", or whatever BOXNAME value other than "refugerestrooms". Then when you run vagrant up, it should create a brand new virtual machine, ignoring the old virtual machine.
• (Optional): To use an existing virtual machine brought up from a different refugerestrooms folder, copy the .vagrant subdirectory of the original refugerestrooms folder into the new one (overwrite the files in the newer folder). That way vagrant up will work from either folder. This actually negates the need for the .box file, since you have an already-provisioned vm, which defeats the purpose of the .box file.
• (Optional): Delete the downloaded box file, if you need the hard-drive space. Vagrant has its own copy of the box file now.

So, I have a box file, but it's not set up to be downloaded automatically. That would involve making/hosting a "catalog", (a small json file) and specifying its URL in the Vagrantfile, and then properly hosting the catalog and the box file on a file server somewhere on the internet. I have tried simulating proper hosting by putting the .box file on Dropbox, and by putting the catalog on GitHub, but it didn't really work; Vagrant gets tripped up with errors on both of these. Neither of them very much likes being a general-purpose file server, and that's probably against their Terms of Use anyway.

(GitHub sends the catalog in a weird way that Vagrant can't figure out is a .json file... and the Dropbox URLs all either have redirects that Vagrant can't figure out how to navigate, or session/authentication mechanisms that end up blocking Vagrant. Google Drive is the same about this as Dropbox is.)

So we would need to probably get something like AWS to make it work, or pay for a remote virtual/dedicated server, or have one of us host our own server at home.

[DeeDeeG]

Here's how I've been making the .box file:

Clone the repository:

• git clone https://github.com/RefugeRestrooms/refugerestrooms

Get the vm provisioned:

• cd refugerestrooms
• vagrant up

SSH into the vagrant box: vagrant ssh

Delete unnecessary software (I found that these things take up a lot of space):

• sudo apt purge juju juju-core
• sudo apt purge libicu52
• sudo apt purge libllvm3.4
• sudo apt purge libsane libsane-common
• sudo apt purge fonts-dejavu-core fonts-ubuntu-font-family-console

(Optional: Use ncdu to find software packages to remove:)

• sudo apt install ncdu
• cd /
• ncdu
• find something taking up space using ncdu
• press q to exit ncdu
• remove stuff with sudo apt remove [package] or with rm -rf [/path/to/thing]

(And when you're done with ncdu, remove it:)

• sudo apt purge ncdu

Clear the cache for the software/package manager (APT), (takes up lots of space):

• sudo apt-get autoremove --purge
• sudo apt-get clean

Make sure users will be able to SSH into your vagrant box (reset the SSH key to the default one):

• cd ~
• rm .ssh/authorized_keys
• wget --no-check-certificate https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub -O .ssh/authorized_keys
• chmod 0600 .ssh/authorized_keys
• chown -R vagrant .ssh

Clear a lot of space, by making every empty bit of the virtual hard-drive a zero (data that's all zeroes in a row can be compressed to almost nothing, and this saves a ton of filesize, since the .box file contains a compressed file system):

• sudo dd if=/dev/zero of=/EMPTY bs=1M wait a while
• sudo rm -f /EMPTY

Turn off and exit the VM:

• sudo shutdown -h now

Make the .box file from the machine we were just tidying up:

• vagrant package --base refugerestrooms --output refugerestrooms.box

(Hint for how to format this command:)

• vagrant package --base --output

[DeeDeeG]

Hard to believe this is so fast and easy with RVM/Docker! Thanks!

(Haven't tried Docker yet, but looking forward to it! I saw in the Travis log that it just downloads a binary, pre-built ruby!)