Initial Heat integration

tomassedovic commented 11 years ago

This is the first step towards full Heat integration. This routes deployment launching and querying for status to Heat. Launch-time parameters should just work, ditto for deleting the instances.

This is mostly to give you guys a direction where I think this should go and to get comments and feedback. I don't think it's wise to merge it just yet -- we need to test it more, clean up the code, find the subtle things this breaks, etc.

There are also a few issues that need fixing:

Heat's API supports atomic launch (i.e. rollback) but as far as I know, it's not been implemented yet
There's a bug in Heat that means the actual instance status isn't reported back correctly. If you launch a deployment and stop the instances out of band, Heat will show them as running.
Heat generates its own wealth of events that we should show in Conductor. They're currently being ignored.

There's also an issue of performance. The way it's implemented now, every time you ask a deployment or an instance for a status, they request the Heat API for an up-to-date value.

This happens a lot of times and it crawls the entire UI into a halt.

I tried caching the value for the lifetime of the object as a simple workaround for rendering, but apparently, we (or Rails) are instantiating the same Deployment and Instance model objects over and over again when rendering the view so this didn't help much.

The same issue is exists with a database backend as well, but Rails transparently caches the DB queries for each controller action.

As a band-aid, I hooked Heat::heat_request to the Rails cache which deals with the issue.

Eventually though, we need to think of a better way to solve it -- either by reducing the number of instances we create or by introducing a proper HTTP caching layer.

Heat's API currently doesn't have any querying capabilities, so to list a few deployments, you have to send a separate request for each and then another request per instance to get instance properties as well. We could work with the Heat community to extend the API in a way that would mean issuing fewer requests.

Before we commit to any optimizations though, it's better to have the raw thing so that we can measure where the bottlenecks actually lie. The advantage of using the built-in Rails caching API here is that you can disable it in the app config without changing anything in the code.

Try it out

You need Heat running with Deltacloud as a backend. I'll write a proper guide tomorrow (I need to package a couple of Python libraries to make things easier).

If you're adventurous, you can go ahead and follow the respective guides:

You can get Heat from here: https://github.com/openstack/heat

No need to set up OpenStack, we're not using it. Though Heat does use a few OpenStack libraries for dependencies.

Next, install deltacloud_heat which is a Deltacloud backend for Heat. It in turn uses the Deltacloud Python client.

https://github.com/tomassedovic/deltacloud_heat

https://github.com/tomassedovic/deltacloud

There were some bugs fixed in the Deltacloud API that we require, you need at least version 1.1.0. We also need the latest Python client which I didn't get to submitting upstream yet. In short, use the server and the client from my fork above.

To configure Heat to use Deltacloud, open the /etc/heat/heat-engine.conf file and add the following line at the end:

cloud_backend=deltacloud_heat.clients

Then open /etc/heat/heat-api.conf and uncomment the last two lines:

[paste_deploy]
flavor = custombackend

Start the Heat engine and api services (if you're going to use the Audrey features, start cloudwatch and cfn-api as well).

That should be it.

martinpovolny commented 11 years ago

A note from IRC discussion: We do not have to pull deltacloud from https://github.com/tomassedovic/deltacloud, but we have to pull python client from the python-client branch in that repo until you create a python package from the python dc client.

tomassedovic commented 11 years ago

Addition: when you run pip install deltacloud_heat, it and the python deltacloud client will be installed. No need to checking out my deltacloud branch any longer.

martinpovolny commented 11 years ago

Thx! The pip install worked as expected.

martinpovolny commented 11 years ago

I know that you said that you do not want to do 'early' optimization. But I really don't like the fact that the head communication is done in the execution path of the request, that is as you said the web interface will freeze, the API call will take ages etc.

I really think that we have to decouple the communication logic from the state information in the database. The API and web UI should just:

display status information and
pass requests (commands) to some backend component some tracker or whatever we call it.

(Sure, there's a 3rd task, that is the 'configurtion management' the updating querying things such as pools, catalogs, strategies.)

I don't think we can fire a bunch of HEAT request to while the user or REST api consumer waits for the response.

We need to queue a request and return it's handle and or just read the state from the database.

Sure, we can create a branch with this code and experiment there, but the performance issue would be IMHO crucial and would result in a NACK.

I hope I understand it right and I suggest that we sit and discuss this face2face tomorrow if you have room for that.

jprovaznik commented 11 years ago

Hi, it looks like a nice start of Heat integration. One technical question: What is the plan about getting instance/deployment state?

From the patch it seems that the state will be fetched per-request, also from the perfomance issue solution (rails caching) it seems you prefer to keep this per-request fetching. This doesn't play nicely with some other entities of conductor which are hooked to the state changes (off-hand: quota, state times, events)

On the other side, dbomatic script was kept so instance/deployment state update is still being executed. Although the state is not used, but for example some timelast*attrs or ip address are used.

Wouldn't it be better for the first cut to just update dbomatic script to do state checking against Heat instead of Deltacloud? This solves your performance issue and all issues with hooked entities. Of course some kind of callbacks/messaging would be much better replacement of dbomatic as discussed many times before, but this is out of scope of this first cut.

tomassedovic commented 11 years ago

Set up instructions

I strongly recommend you do this in a VM. Tested on a Fedora 18 JEOS.

Couple of notes:

The Deltacloud and Heat versions packaged in Fedora don't have the necessary fixes so they need to be installed from source. Once the newest version are packaged, a lot of the steps here will go away.
Openstack (and by extension, Heat) uses MySQL by default. It should work with PostgreSQL, but I didn't look into it yet. That's one of the next things on my list (unless someone more experienced wants to pick it up).

Dependencies

$ sudo yum install heat git MySQL-python ruby-devel rubygem-sqlite3 \
  qpid-cpp-server python-crypto gcc ruby-devel libxml2 libxml2-devel \
  libxslt libxslt-devel gcc-c++

a lot of these are dependencies for building the native extensions for gems and python libs we need. Once the latest Heat and Deltacloud versions get packaged, this will just read: yum install heat deltacloud-core

Install Heat from the source

$ git clone git://github.com/openstack/heat.git
$ cd heat
$ sudo python setup.py install
$ sudo heat-db-setup rpm -y -r ''
$ sudo cp etc/heat/* /etc/heat/

nothing special about the Heat commit. My installation has it so we'll be on the same page when debugging issues

Deltacloud backend for Heat

Install the deltacloud_heat package:

$ sudo easy_install deltacloud_heat

Open /etc/heat/heat-api.conf and add the following lines to the end:

[paste_deploy]
flavor = custombackend

Find the IP address of the machine with Heat installed:

$ ip addr

Open /etc/heat/heat-engine.conf and add the following line to the end:

cloud_backend=deltacloud_heat

Find and replace 127.0.0.1 with the actual IP address in: heat_metadata_server_url, heat_waitcondition_server_url and heat_watch_server_url.

So, if your IP address is 10.0.2.15, this is what it should look like:

heat_metadata_server_url = http://10.0.2.15:8000
heat_waitcondition_server_url = http://10.0.2.15:8000/v1/waitcondition
heat_watch_server_url = http://10.0.2.15:8003

Make sure you have Deltacloud server version 1.1.0 or higher:

$ sudo gem install --no-ri --no-rdoc deltacloud-core

Running the services

Run the deltacloud server.

$ deltacloudd -i mock

Start the message bus (Heat services use it for internal communication):

$ sudo systemctl start qpidd

Start Heat services:

$ sudo heat-engine
$ sudo heat-api
$ sudo heat-api-cfn
$ sudo heat-api-cloudwatch

depending on your preferences you may start the services on the background or in different ssh/screen sessions. You can open the ports 8000-8004/tcp if you want to access Heat from the outside.

Test Heat/Deltacloud integration

$ curl -v -H 'accept: application/json' -H 'x-auth-user: mockuser' -H 'x-auth-key: mockpassword' \
  -H 'x-auth-url: http://localhost:3001/api' -H 'x-roles;' \
  http://localhost:8004/v1/deltacloud/stacks

This should return a HTTP 200 response with {"stacks": []} as the body.

Note that X-AUTH-USER header expects the Deltacloud username, X-AUTH-KEY Deltacloud password and X-AUTH-URL Deltacloud API entry point.

Conductor

Conductor should work normally (with the pull request applied). You set up the Heat API entry point in src/config/settings.yml but the default should work if you install Heat and Conductor side by side.

aeolusproject / conductor