search

amplab / training-scripts

Scripts to launch cluster used for Strata
33 stars 34 forks source link

Updating script for admin scripts moving to sbin #10

Open patmcdonough opened 10 years ago

shivaram commented 10 years ago

Hey @patmcdonough -- Do you know what Spark version was used in the sparksummit1 branch ? I just want to make sure that the version on the AMI has sbin instead of bin

patmcdonough commented 10 years ago

Hey @shivaram - good question. @pwendell and I were talking about that earlier as I hadn't realized the latest branch was ampcamp4 (although that's obvious in hindsight).

I did build a cluster earlier off of the sparksummit1 branch and that's when I found this change was required. But I could have sworn we ran Spark Summit off of v0.8. Is the script downloading the latest version somehow, or maybe it's using a newer AMI?

shivaram commented 10 years ago

It gets the AMI from http://s3.amazonaws.com/ampcamp-amis/latest-ampcamp3 -- I am pretty sure all the AMPCamp and Spark Summit clusters have used a pre-installed Spark instance on the AMI and don't download anything during launch.

I am not very sure, but did the S3 symlink changed sometime after the summit ?

patmcdonough commented 10 years ago

Ah, that explains it I think. I just went to the AWS console and searched for that AMI (ami-19474270 right now). The title of the AMI is "ampcamp4-4," so I'm guessing that symlink changed inadvertently.

It probably makes the most sense to close this PR if we can fix the symlink so this correctly represents the snapshot from Spark Summit, and so the online training docs will still work.

shivaram commented 10 years ago

Okay. Do you or @pwendell have the old AMI ID handy somewhere ? I can update it in S3, but don't know what the value should be.