= Merlin
== What is Merlin?
Merlin is a Ruby on Rails web application for managing and controlling EC2 API-compatible clouds (Eucalyptus / AWS EC2). The primary goal of the project is to provide an EC2-like interface to Eucalyptus private clouds; however, compatibility with both is a primary goal (and EC2 functionality has been tested). Support is built-in for managing multiple clouds with different credentials, endpoints, etc.
Merlin's main goal is to provide a flexible, "pretty" UI for managing private and public cloud resources, as well as providing highly usable Rails objects which can be used to interact with the cloud in any number of ways.
Merlin also, as of this writing, provides services such as auto-provisioning (using Puppet), interfaces to more detailed object information (such as hostname, function, etc), automatic DNS updating (BIND and UltraDNS are currently supported) and more.
Merlin also exposes a simple HTTP API that can be used by an instance to perform various functions (such as attach storage, introduce itself, etc). Merlin is intended to allow a systems administrator to quickly see what machines are in their cloud and what they are doing, along with giving the administrator the ability to quickly spin up new instances, get coffee, come back, and start using them.
Eventually, it is my hope that Merlin will expand to have the functionality of many proprietary cloud management services (e.g. RightScale), and will allow a user to click a few times and create an entirely new cluster of instances with appropriate volumes / load balancers created. Work is ongoing to allow Merlin to use the F5 (http://www.f5.com) API to autoregister new webheads with a load balancer, and ELB support is also planned. This means that you will be able to launch new instances with Merlin with a few clicks; these instances will then launch, use Merlin and Puppet to autoprovision themselves, contact Merlin when they are ready to be used, and then can be automatically assigned to a load balancer.
That's neither here nor there, however. As to what exists today...
First, a warning: this is a very new wproject, and progress is ongoing. The views are simplistic and very ugly, but they work. You should know that this is my first Ruby project of any kind; this may show itself in the quality of the code, for which I humbly apologize. However, if you don't like it, you should send some patches. Obviously.
== Currently Implemented Functionality
Today, Merlin can:
Merlin can create new cloud instances, and if you use the default user data script (with some modifications, see "Provisioning & Merlin"), the instances will report back to Merlin when they are active, and after they finish provisioning (currently, only Puppet is supported, feel free to send patches for other systems). Each instance upon creation, is assigned an access token which can be used by that instance to call a number of API functions (see "API Functions"). These API functions are simplistic at the moment - simple GETs via HTTPS - and they allow the instance to indicate to Merlin that it is up and running, that it has finished provisioning, or that an error has occured which requires user intervention.
Merlin has support for updating BIND and UltraDNS DNS servers automatically. Once an instance introduces itself to Merlin using the "hello" API call, Merlin can immediately register either its private or public addresses with a DNS provider.
Merlin can create new volumes, and attach them to instances. Also, if you use the default userdata functionality, upon startup, an instance can request that Merlin connect its storage. The user data script will then set up this storage according to your settings, formatting & mounting as necessary.
Basic functionality is supported for taking snapshots of volumes. There will be further support in the API for doing this programatically using the instance's access_token.
Merlin is intended to be able to poll clouds regularly, which will update Merlin's rails objects to mirror the following: key pairs, availability zones, instances, volumes, and snapshots. This allows you to "import" clouds, so that Merlin can work with existing setups.
Merlin provides a lot of neat functionality in its models, which can also make it very handy as the basis for more intelligent command line tools (documentation is forthcoming, but here are some simple examples).
ec2 = Cloud.find_by_name "Amazon EC2 (us-east)"
ec2.update_from_api
ec2.object_exists? i-13371337
ec2.instances.all
AvailabilityZone.find_by_name('us-east-1a').instances.all
it = InstanceType.new it.name = "My Ubuntu Lucid 10.04 Image" it.image_id = "ami-13371337" it.kernel_id = "aki-DEADBEEF" it.vm_type = cloud.vm_types.find_by_name('m1.small') it.bits = 64 it.description = "My custom Ubuntu Lucid image." it.os = "Ubuntu Lucid 10.04" it.save
i = Instance.new i.cloud = Cloud.find_by_name 'Amazon EC2 (us-east)' i.hostname = 'foo-01.bar' i.instance_type = InstanceType.find_by_name_and_bits('My Ubuntu Lucid 10.04 Image', 64) i.key_pair = KeyPair.find_by_name('mycloudkey') i.security_group = SecurityGroup.find_by_name('default-security-group') i.user_data = Userdata.find_by_name('My Userdata Script') i.reserve
i.reservation_id i.status_code i.status_message
i.availability_zone.name
v = Volume.new v.size = 10 v.availability_zone = i.availability_zone v.reserve
v = Volume.new v.snapshot_id = "snap-13371337" v.availability_zone = i.availability_zone v.reserve
== Sharp Edges
Merlin is missing a lot of stuff: model validations, sanity checks, etc. All of this will be added. Merlin will definitely let you not only shoot yourself in the foot, but if you're a Eucalyptus user, and you give Merlin an admin account, it will also let you blow everyone else's foot away as well. You have been warned, be careful with it. There are no protections against doing unbelievably stupid things with Merlin....yet.
== Getting Started
== Requirements
Gems
Gem requirements are detailed in the Gemfile.
Message Server
Merlin requires a STOMP-compatible message broker, and the ActiveMessaging library (included). I recommend ActiveMQ: http://activemq.apache.org. Setting up and running ActiveMQ is as easy as running bin/activemq from an unpacked tarball.
Configuration is in config/broker.yml.dist, and should have sane defaults which will let you get started easily, assuming AMQ is on the local machine.
script/poller
The poller script, for ActiveMessaging processors, needs to be running whenever Merlin is.
== Copyright / Licensing Information
Merlin - the only limit is the sky Copyright (C) 2011 Justin Baugh, baughj@discordians.net
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
== Additional Resources
=== On the Web
== Credits
Thanks to Matt Lee for providing the Merlin logo, and the clever tagline.
== Contact
If you're interested in hacking on Merlin (and I hope you are), I am happy to receive requests on either Gitorious or Github. You can also just send an email to mailto:baughj@discordians.net or join the Google Groups forum.
== Patches & Pull Requests
Here's how I hope you'll do this:
Fork the main Merlin repository, on either Gitorious or Github.
Create a branch that describes what you're doing: git checkout -b fixing_amazingly_terrible_bugs_holy_crap
Please don't use "master" or "development" in your fork.
Keep your changesets small, don't make a commit that changes eleventy billion things or can't be easily merged into master.
Feel free, though, to just send patches.