Error during installation

henryngo commented 6 years ago

ubuntu@ip-123-45-67-789:~$ ./node_modules/.bin/spotswap-install /home/ubuntu/node_modules/@mapbox/spotswap/index.js:18 if (!process.env[key]) throw new Error('Env var ' + key + ' is required'); ^

Error: Env var INSTANCE_ID is required at /home/ubuntu/node_modules/@mapbox/spotswap/index.js:18:34 at Array.forEach (native) at envCheck (/home/ubuntu/node_modules/@mapbox/spotswap/index.js:17:5) at Object. (/home/ubuntu/node_modules/@mapbox/spotswap/bin/install.js:9:1) at Module._compile (module.js:410:26) at Object.Module._extensions..js (module.js:417:10) at Module.load (module.js:344:32) at Function.Module._load (module.js:301:12) at Function.Module.runMain (module.js:442:10) at startup (node.js:136:18)

What are the actual steps to install and deploy this to an existing AWS environment? There seem to be a lot of missing info in the documentation. I'm willing to help once I have it up and running. Thanks.

cc: @arunasank @emilymcafee @jakepruitt

cristim commented 6 years ago

@henryn001 sorry for this shameless plug on your issue, but you should have a better getting started experience using my similar project https://github.com/cristim/autospotting

henryngo commented 6 years ago

@cristim I have your project up and running. It does work well however, we need something that has the termination logic that will spin up the back-up on-demand instances once a spot instance has been tagged for termination.

cristim commented 6 years ago

@henryn001 thanks, I'll definitely consider this idea for implementation soon, it's on the roadmap for a lot of time.

In the meantime you can do it relatively easily with an event that is notified for spot termination that calls a Lambda function that detaches the spot instance from the group with the capacity replacement flag set in the detach API call.

jakepruitt commented 6 years ago

Word @henryn001 - appreciate the feedback (also +1 for feedback and collaboration @cristim - we're thinking of using cloudwatch events in the near future too).

It looks like the EnvironmentVariables section is lacking and needs a few additions:

AWS_REGION: the region of the instance
INSTANCE_ID: the ID of the instance. We get this by using the metadata endpoint at curl -s -m 3 http://169.254.169.254/latest/meta-data/instance-id.
SpotGroup or SpotFleet: the ID of the spot autoscaling group or spotfleet. We get this by making a cloudformation API call to describe the stack the instance is a part of, and query for the physical resource ID, with something like this:

asg=$(aws cloudformation describe-stack-resource \
    --stack-name ${stack} \
    --logical-resource-id AutoScalingGroup \
    --query StackResourceDetail.PhysicalResourceId \
    --output text)

I'll cut a PR to add this to the readme. Thanks for the catch!

henryngo commented 6 years ago

@jakepruitt Thanks for the update. I'm now getting an "Error: EACCES: permission denied, open '/etc/init/spotswap-poll.conf'". Also, is INSTANCE_ID for an instance (any) that's not part of the scaling group? More like an Admin box?

I'm more of an Ops than a Dev so perhaps I'm approaching this the wrong way. Is there a CloudFormation template I can use and a Lambda function zip I can just upload?

jakepruitt commented 6 years ago

INSTANCE_ID should be the ID of the instance that spotswap is currently running on, pulled from the metadata endpoint http://169.254.169.254/latest/meta-data/instance-id during a UserData script run on an instance. An example UserData script you'll probably need should look something like this (you'll need to clone your repo into ${deploy_dir} on the instance, and set ${stack} as the AWS::StackName):

instance=$(curl -s -m 3 http://169.254.169.254/latest/meta-data/instance-id)
asg=$(aws cloudformation describe-stack-resource \
    --stack-name ${stack} \
    --logical-resource-id AutoScalingGroup \
    --query StackResourceDetail.PhysicalResourceId \
    --output text)
SpotGroup=${asg} \
terminationTimeout=0 \
INSTANCE_ID=${instance} \
AWS_REGION=${region} \
${deploy_dir}/../node_modules/.bin/spotswap-install

I'm not sure about the user permissions of the UserData script, but I assume it already is or can be configured to be a user that has access to the /etc/init directory, which sounds like the problem you ran into.

In terms of the lambda function zip and cloudformation template, you'll need to write your own template in javascript and include the @mapbox/spotswap module as a npm dependency (Check out the usage section of the readme for an example of the things you'll need in your template).

In terms of lambda bundles, I suggest zipping up the repo of the code you're planning on running for the cloudformation stack (with the node_modules installed, and @mapbox/spotswap defined as a dependency) and uploading it to S3. At Mapbox, we use https://github.com/mapbox/stork/ to use CodeBuild to build our github repositories into zipped bundles on S3 (if you're doing anything with lambda, I'd definitely recommend taking a look at this from a devops standpoint).

mapbox / spotswap

Error during installation #8