IaaS-cloud backend for GC3Pie

[GoogleCodeExporter]

The goal/idea is to have a mechanism to access a computational
resource that is entirely existing in a private/public cloud
(EC2-compliant) and present it as a regular GC3Pie LRMS.

There are basically two proposals.  We may choose to implement one or
the other, or, possibly, both. 

Proposal (A)
============

Have a GC3Pie "virtual cluster" LRMS that, like StarCluster[1] can
activate a computational cluster on demand (let's say running SLURM)
and be used/accessed through regular SSH.  In the perspective of
developing a Data Management module for GC3Pie, it should also be
possible to attach a cloud storage (S3 compliant) for data processing.

[1]: http://web.mit.edu/star/cluster

Proposal (B)
============

Forget about batch-queueing systems, and rather model
the "cluoud backend" on the "localhost+ssh" backend.  In more details:
the "cloud backend" will spawn VMs on a certain infrastructure, and
then we connect to each VM via SSH and launch the needed application.

An example configuration stanza might be this one:

    [resource/ec2]
    type = libcloud
    driver = ec2-us-east
    access_key = ...
    secret_key = ...
    max_instances = 100
    gamess_vm = ami-XXXXXX
    rosetta_vm = ami-YYYYYY
    # ...

The above stanza would configure the backend for connecting to
Amazon's EC2 using the given access keys.  The VM template to use is
read off the "<application>_vm" key; e.g., for running GAMESS,
instances of AMI-XXXXXX would be used.

No more than 100 (`max_instances`) VMs will ever be spawned
concurrently; if a job is submitted when there are already
`max_instances` VMs running, the submission process raises
`LRMSSubmitError` (like `localhost` does when `max_cores` is
exceeded).

Original issue reported on code.google.com by riccardo.murri@gmail.com on 2 Nov 2012 at 4:18

• Blocked on: #366

[GoogleCodeExporter]

Original comment by riccardo.murri@gmail.com on 10 Jan 2013 at 2:05

[GoogleCodeExporter]

Original comment by riccardo.murri@gmail.com on 10 Jan 2013 at 2:14

• Now blocked on: #366

[GoogleCodeExporter]

This issue was closed by revision r3229.

Original comment by antonio....@gmail.com on 11 Jan 2013 at 4:34

• Changed state: Fixed