islandora-interest-groups / Islandora-DevOps-Interest-Group

The purpose of the Islandora DevOps Interest Group is to make it easier for endusers, developers, testers, and system administrators to understand, deploy, tune and secure the Islandora stack (and associated tools) in their respective hosting environments.
14 stars 17 forks source link

Migration from Vagrant / VB to Docker #12

Closed JacobSanford closed 6 years ago

JacobSanford commented 8 years ago

We are currently transitioning away from development in Vagrant and bare metal/OS deployment. The large overhead, low density, lengthy spin-up time, dependence on external repositories, complicated dev->test->live transitions, and general persnicketiness of Vagrant has been a 'elephant in the room' of our development pipeline for some time. We are a small team and we were spending a lot of time paying down technical debt to perform some simple tasks.

Moving towards a container based pipeline was our response. As a base container system, RkT isn't currently as mature, doesn't have the momentum of Docker, has a steeper learning curve for new developers, and as a result we've chosen Docker. This change has a big bonus of synchronizing our development environments almost perfectly with production, as we are also transitioning away from multiple VMWare VM Ubuntu LTS instances towards a more streamlined CoreOS cluster with GlusterFS backed persistent storage.

Anyhow, we've got our fedora-solr-gsearch docker image to a point where I am comfortable (read: not completely embarrassed) with putting it 'out there' for feedback and community improvement/hackery/feedback. On its own, the image isn't immediately useful for others, but coupled with the docker-islandora-dev repository, it becomes a reasonable, fast and extensible Development pipeline foundation for Islandora. We are using it for development now.

The docker-islandora-dev repo is largely based on our drupal image that we use in production and leverages phusion/baseimage's my_init system to ease image maintenance.

The fedora-solr-gsearch certainly isn't production-ready, but I think with some further work it can be polished up nicely. If you're new to container infrastructure, I have hammered out a crude wiki that can serve as a quickstart guide for Islandora.

tl;dr - Docker islandora development at docker-islandora-dev. Instructions/wiki at https://github.com/unb-libraries/docker-islandora-dev/wiki. Feedback welcomed. Once it is polished up a bit further, we would like to send it to the general Islandora list.

manez commented 8 years ago

Ping me when you're ready to go more public with this. I'd like to help promote!

pc37utn commented 8 years ago

Will the vagrant devel be supported anymore? We are still in the process of getting people to use vagrant and actually still have an order placed through our library for a book about vagrant.... it may be here in 3 months. :) Paul C UT Libraries

On 09/01/2015 02:24 PM, Melissa Anez wrote:

Ping me when you're ready to go more public with this. I'd like to help promote!

— Reply to this email directly or view it on GitHub https://github.com/islandora-interest-groups/Islandora-DevOps-Interest-Group/issues/12#issuecomment-136819812.

ruebot commented 8 years ago

I don't plan on going any where :smile:

JacobSanford commented 8 years ago

Paul : Apologies if I triggered any fears : I certainly have no intentions for it to replace the Islandora vagrant development in any way. Since we are moving away from Vagrant in general, docker simply fits us better!

pc37utn commented 8 years ago

No fears! Containers are another thing on our list but they are a little further down. Looking forward to exploring though!

thanks, Paul C UT Libraries

On 09/01/2015 03:49 PM, Jacob Sanford wrote:

Paul : Apologies if I triggered any fears : I certainly have no intentions for it to 'replace' islandora vagrant development in any way. It simply fits our pipeline better!

— Reply to this email directly or view it on GitHub https://github.com/islandora-interest-groups/Islandora-DevOps-Interest-Group/issues/12#issuecomment-136839168.

ruebot commented 8 years ago

...and it's good to have this as an option, and I can't be happier to see it. I went down this path a few years ago and abandoned it: https://github.com/ruebot/docker-islandora :smile:

daniel-dgi commented 8 years ago

You've certainly pique my interest on this. We're using vagrant and bash scripts for the 7.x-2.x project, and while I love having that dependable environment, destroying and spinning up is time consuming (especially when you live out in the boonies with ancient DSL interwebs). Hopefully there's some lessons to learn from this!

pc37utn commented 8 years ago

"destroying and spinning up"? maybe I've been using it wrong. I save snapshots and upgrade inside the snapshot... :)

I have really enjoyed the vagrant scripts. I took them apart and made a client install to throw on some spare servers so large ingests wouldn't slow down the site. Haven't fully tested it yet... adding different users and links to our NFS, etc.

but anyway, it all is great!

On 09/01/2015 03:55 PM, Daniel Lamb wrote:

You've certainly pique my interest on this. We're using vagrant and bash scripts for the 7.x-2.x project, and while I love having that dependable environment, destroying and spinning up is time consuming (especially when you live out in the boonies with ancient DSL interwebs). Hopefully there's some lessons to learn from this!

— Reply to this email directly or view it on GitHub https://github.com/islandora-interest-groups/Islandora-DevOps-Interest-Group/issues/12#issuecomment-136840875.

daniel-dgi commented 8 years ago

Oh, I always give it a full run when testing the install. I'm paranoid like that.

ksclarke commented 8 years ago

@daniel-dgi One way to reduce the downloading and spinning up time with vagrant is to use Packer.io to do the build and output a vagrant box from that. Yes, there is still the downloading and spinning up when you do the Packer build, but when you just want to run your vagrant box it's already all on there for you.

/me takes off packer.io evangelist hat and gets back to work

(I know at least some of the islandora vagrant builds cache the files locally too, but taking any chance I can to evangelize).

ruebot commented 8 years ago

@ksclarke do we have documentation for how to do that? If not, I think that'd be a wonderful contribution to islandora_vagrant and/or Islandora 7.x-2.x vagrant.

JacobSanford commented 8 years ago

:+1: To packer.io from me as well.

ksclarke commented 8 years ago

@ruebot Well it's a bit of a different config/project. You're just outputting the vagrant box as a build artifact rather than using vagrant to do the build so it's packer.io actually running your bash scripts... in short, it's a different setup than the current islandora_vagrant builds. You could use the same bash scripts (for the most part), but the project setup would vary from what it is now.

Edit: And I agree it would be a wonderful contribution... I don't have the bandwidth at the moment to tackle it though.

JacobSanford commented 8 years ago

@daniel-dgi I could throw together a 2.0 branch in the next few weeks if you were interested.

JacobSanford commented 8 years ago

@ruebot Without much fuss, Vagrant also has a 'package' option that could be used in a build process to snapshot an Islandora box after it has been built. One could be released along with each version (and Jenkins could re-build it on a regular basis to update package security, etc).

The box wouldn't be as useful for deployment, but would meet (by my perception, at least) a significant portion of the user/development need. The Vagrantfile could then be simplified into:

The vagrant would then trade up-front downloading to shed a ton of time and network weight when doing 'vagrant up', since you're only doing a Drupal update after boot.

ruebot commented 8 years ago

...a 7.x-2.x branch would be wonderful. We'd really love collaboration between DevOps and Fedora 4 IG :smile:

@JacobSanford sounds like exactly what I was getting at with https://github.com/Islandora-Labs/islandora_vagrant/issues/27 -- this would be a great sprint project!

g7morris commented 6 years ago

Closing. Can reopen if group / individual decides to pursue topic.