Need documentation on install and killing vagrant

[wmburke]

Just like in the deal...

during the hackathon

[ahmadia]

+1

[ahmadia]

• [ ] - Download/install Vagrant. Vagrant version needs to be 1.7 or greater
• [ ] - Run vagrant up

[ahmadia]

Some notes from this email, need to be turned into real documentation.

Downloading and provisioning Memex Explorer takes about an hour on a fast Internet connection (~ 1 MB/s or better). The biggest cost is installing three Docker containers for Tika, Kibana, and ElasticSearch. You can trigger an installation with the following commands:

git clone git@github.com:memex-explorer/memex-explorer.git
# https://github.com/memex-explorer/memex-explorer.git for the HTTPS folks
cd memex-explorer
vagrant up

This has been much more thoroughly tested than the version we had ready on Thursday.

I've enabled verbose output from the Salt calls, so you should see much more output than the previous installer. It is a good idea to monitor your network connection if you want to see if the build is stalled. An idle network connection and an idle console may mean that you need to interrupt/restart.

Once the install is complete, you should eventually see some output like this in your log:

Summary
-------------
Succeeded: 38 (changed=9)
Failed:     0
-------------
Total states run:     38
==> default: Running provisioner: app (shell)...
    default: Running: inline script
==> default: stdin: is not a tty
==> default: MEMEX-EXPLORER IS NOW RUNNING, POINT YOUR BROWSER TO: http://localhost:8000

You can now launch Memex Explorer. If you open a browser at http://localhost:8000 you should be able to create a new project, create a new crawl, and run the crawl with verbose log output via log.io. I am working on getting ElasticSearch working with the Salty Vagrant, but this functionality is currently broken.

Don't hesitate to email the list or raise an issue at https://github.com/memex-explorer/memex-explorer/issues if you run into any issues or failures in the above process. If you'd like to work from an existing repository instead of a fresh clone, it's imperative that you delete any potential conflicting files:

cd memex-explorer
git fetch origin
git checkout origin/master
cd source
git clean -xfd
cd ..
vagrant reload
vagrant provision

Remember that if you're having issues on the Vagrant box, the easiest thing to do is to kill all Python processes, instead of sending hard interrupts through vagrant:

vagrant ssh
sudo killall python

• A few more details *

The main workhorse here is Salt, which is a configuration management tool that I'm using in a really naive manner (picture somebody programming without using any variables). Salt is operating in "masterless" mode, that is, we're both configuring from/to the same machine.

As a thin shell around Salt, we're using Vagrant (introduce salty vagrant jokes here!). Vagrant is a tool for managing the relationship between hosts (your laptop) and clients (virtual machines running on your laptop). Vagrant does things like map the contents of the repository on the host machine to /vagrant on the client box, forward ports, choose the Ubuntu instance to install, and bootstrap Salt.

The awesome thing about Vagrant is that it can be used to provision local laptops, but it also can seamlessly provision EC2 and OpenStack instances [1][2]. With the appropriate configurations and credentials, the command goes from:

vagrant up

to something like:

vagrant up --provider=openstack

All of the Vagrant configuration lives in a top-level Vagrantfile.

All of the Salt configuration lives in the salt/ subdirectory that lives in the top-level. Both the Vagrantfile and the Salt files are very readable. Vagrantfiles are written in Ruby, and the Salt files are Jinja-templated YAML.

[1] http://docs.aws.amazon.com/opsworks/latest/userguide/cookbooks-101.html [2] https://github.com/cloudbau/vagrant-openstack-plugin

[lewismc]

Very helpful thanks

On Tuesday, May 26, 2015, Aron Ahmadia notifications@github.com wrote:

Some notes from this email, need to be turned into real documentation.

Downloading and provisioning Memex Explorer takes about an hour on a fast Internet connection (~ 1 MB/s or better). The biggest cost is installing three Docker containers for Tika, Kibana, and ElasticSearch. You can trigger an installation with the following commands:

git clone git@github.com javascript:_e(%7B%7D,'cvml','git@github.com'); :memex-explorer/memex-explorer.git https://github.com/memex-explorer/memex-explorer.git for the HTTPS folks

cd memex-explorer vagrant up

This has been much more thoroughly tested than the version we had ready on Thursday.

I've enabled verbose output from the Salt calls, so you should see much more output than the previous installer. It is a good idea to monitor your network connection if you want to see if the build is stalled. An idle network connection and an idle console may mean that you need to interrupt/restart.

Once the install is complete, you should eventually see some output like this in your log: Summary

Succeeded: 38 (changed=9) Failed: 0

Total states run: 38 ==> default: Running provisioner: app (shell)... default: Running: inline script ==> default: stdin: is not a tty ==> default: MEMEX-EXPLORER IS NOW RUNNING, POINT YOUR BROWSER TO: http://localhost:8000

You can now launch Memex Explorer. If you open a browser at http://localhost:8000 you should be able to create a new project, create a new crawl, and run the crawl with verbose log output via log.io. I am working on getting ElasticSearch working with the Salty Vagrant, but this functionality is currently broken.

Don't hesitate to email the list or raise an issue at https://github.com/memex-explorer/memex-explorer/issues if you run into any issues or failures in the above process. If you'd like to work from an existing repository instead of a fresh clone, it's imperative that you delete any potential conflicting files:

cd memex-explorer git fetch origin git checkout origin/master cd source git clean -xfd cd .. vagrant reload vagrant provision

Remember that if you're having issues on the Vagrant box, the easiest thing to do is to kill all Python processes, instead of sending hard interrupts through vagrant:

vagrant ssh sudo killall python

• A few more details *

The main workhorse here is Salt, which is a configuration management tool that I'm using in a really naive manner (picture somebody programming without using any variables). Salt is operating in "masterless" mode, that is, we're both configuring from/to the same machine.

As a thin shell around Salt, we're using Vagrant (introduce salty vagrant jokes here!). Vagrant is a tool for managing the relationship between hosts (your laptop) and clients (virtual machines running on your laptop). Vagrant does things like map the contents of the repository on the host machine to /vagrant on the client box, forward ports, choose the Ubuntu instance to install, and bootstrap Salt.

The awesome thing about Vagrant is that it can be used to provision local laptops, but it also can seamlessly provision EC2 and OpenStack instances [1][2]. With the appropriate configurations and credentials, the command goes from:

vagrant up

to something like:

vagrant up --provider=openstack

All of the Vagrant configuration lives in a top-level Vagrantfile.

All of the Salt configuration lives in the salt/ subdirectory that lives in the top-level. Both the Vagrantfile and the Salt files are very readable. Vagrantfiles are written in Ruby, and the Salt files are Jinja-templated YAML.

[1] http://docs.aws.amazon.com/opsworks/latest/userguide/cookbooks-101.html [2] https://github.com/cloudbau/vagrant-openstack-plugin

— Reply to this email directly or view it on GitHub https://github.com/memex-explorer/memex-explorer/issues/511#issuecomment-105747494 .

Lewis

[brittainhard]

This is in the docs now, see http://memex-explorer.readthedocs.org/en/latest/vagrant_guide.html#cheat-sheet