Instructions on how to build AMIs suitable for StarCluster

[rustandruin]

Apt-get does not work with any of the StarCluster AMIs publicly available, as a consequence of those using versions of Ubuntu that are no longer supported.

It would be great if you could document the process of creating an AMI for StarCluster, so that we could roll our own AMIs with the latest versions of Ubuntu as they come out.

[edechter]

I had the same problem. If you look at the bottom of the starcluster home page it mentions the latest AMIs:

Latest Stable AMI IDs (us-east-1):

ami-7c5c3915 (i386) ami-765b3e1f (x86_64) ami-52a0c53b (HVM+GPU)

ami-765b3e1f uses ubuntu 12.04 which is still supported.

The config file template needs to be updated to tell people to look at that link, and the default should be updated to the most recent versions.

Confusingly, if you run starcluster listpublic you get:

$ starcluster listpublic
StarCluster - (http://star.mit.edu/cluster) (v. 0.95.6)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

*** WARNING - Setting 'AWS_SECRET_ACCESS_KEY' from environment...
*** WARNING - Setting 'AWS_ACCESS_KEY_ID' from environment...
>>> Listing all public StarCluster images...

32bit Images:
-------------
[0] ami-9bf9c9f2 us-east-1 starcluster-base-ubuntu-13.04-x86 (EBS)
[1] ami-7c5c3915 us-east-1 starcluster-base-ubuntu-12.04-x86 (EBS)
[2] ami-899d49e0 us-east-1 starcluster-base-ubuntu-11.10-x86 (EBS)
[3] ami-8cf913e5 us-east-1 starcluster-base-ubuntu-10.04-x86-rc3
[4] ami-d1c42db8 us-east-1 starcluster-base-ubuntu-9.10-x86-rc8

64bit Images:
--------------
[0] ami-3393a45a us-east-1 starcluster-base-ubuntu-13.04-x86_64 (EBS)
[1] ami-6b211202 us-east-1 starcluster-base-ubuntu-13.04-x86_64-hvm (HVM-EBS)
[2] ami-52a0c53b us-east-1 starcluster-base-ubuntu-12.04-x86_64-hvm (HVM-EBS)
[3] ami-765b3e1f us-east-1 starcluster-base-ubuntu-12.04-x86_64 (EBS)
[4] ami-4583572c us-east-1 starcluster-base-ubuntu-11.10-x86_64-hvm (HVM-EBS)
[5] ami-999d49f0 us-east-1 starcluster-base-ubuntu-11.10-x86_64 (EBS)
[6] ami-0af31963 us-east-1 starcluster-base-ubuntu-10.04-x86_64-rc1
[7] ami-2faa7346 us-east-1 starcluster-base-ubuntu-10.04-x86_64-qiime-1.4.0 (EBS)
[8] ami-8852a0e1 us-east-1 starcluster-base-ubuntu-10.04-x86_64-hadoop
[9] ami-a5c42dcc us-east-1 starcluster-base-ubuntu-9.10-x86_64-rc4
[10] ami-06a75a6f us-east-1 starcluster-base-centos-5.4-x86_64-ebs-hvm-gpu-hadoop-rc2 (HVM-EBS)
[11] ami-12b6477b us-east-1 starcluster-base-centos-5.4-x86_64-ebs-hvm-gpu-rc2 (HVM-EBS)

total images: 17

which suggests that ami-3393a45a or ami-6b211202 are more recent. But I think that these aren't supported because they are not designated by Ubuntu as Long Term Releases .

[rustandruin]

https://github.com/rustandruin/starcluster-spark-skylark has a solution for generating a StarCluster compatiable AMI off Ubuntu 15.04 that worked for me.

[andrewcstewart]

It would be nice to come up with a way to turn -any- AMI into a StarCluster node, and not just one specifically pre-built to the task. I've read before that this might be possible by just using the SGE plugin over the default SGE provisioner, but that doesn't seem to take care of things like NFS.

Maybe something like https://www.packer.io could be of use as a StarCluster AMI builder.

[cancan101]

TBH I have been pretty happy with:

sudo apt-get install -y rpcbind nfs-common nfs-kernel-server rxvt

and https://github.com/cancan101/StarCluster/commit/be92320e020e7fc07d5915546f424c44c30324fa, along with just making sure that sge is installed to /opt/sge6-fresh.

[andrewcstewart]

@cancan101 Cool, so that's the absolute minimal requirements. So what I might do when i get a minute is throw that into a packer recipe (might need to generalize it for non-debian distros). That should let someone StarClusterify most any existing AMI.

[cancan101]

@andrewcstewart https://github.com/jtriley/StarCluster/pull/435 is probably better than my commit.

[cancan101]

Though that PR only deals with one of the service name changes.

[clearf]

@andrewcstewart, curious if you've made any progress getting a recipe rolled out?

[andrewcstewart]

@clearf A basic Packer recipe would look like this:

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": "",
    "source_ami": ""
  },
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "region": "us-east-1",
    "source_ami": "{{user `source_ami`}}",
    "instance_type": "t2.micro",
    "ssh_username": "ubuntu",
    "ami_name": "StarCluster AMI {{timestamp}}"
  }],
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sleep 30",
      "sudo apt-get update",
      "sudo apt-get install -y nfs-kernel-server nfs-common portmap",
      "sudo apt-get install -y gridengine-client gridengine-common gridengine-master gridengine-qmon sun-java6-jre"
    ]
  }]
}

And then be executed with Packer via (assuming the above is saved to starclsuter-ami.json:

packer build -var 'aws_access_key=XXX' -var 'aws_secret_key=YYY' -var 'source_ami=ZZZ' starcluster-ami.json

...though I haven't tested it, and I'm not sure what from the PR @cancan101 referred to needs to be ported into the recipe's provisioner (vs simply being part of an updated StarCluster distribution). Feel free to play around with that as a starting point.

[xanderdunn]

Here are some instructions I wrote out: https://medium.com/p/bdd32245a0d9

I simply started with the base Ubuntu 14.04 AMI provided by Amazon.

[rustandruin]

Thanks Xander; your instructions look straightforward (haven't tried them, though). The link you provided ends in `/edit'. You should probably remove that.

[rubenvereecken]

@xanderdunn I followed pretty much exactly your steps during my attempts to get this working, with the difference that it ends up not working. Executing qhost on my master I usually see the nodes but don't get any stats on them. When scanning ports I see there's no actual connection established. Haven't figured out why this won't work yet. Starting anew isn't really an option because of the time invested in creating this AMI.