2020-01-27 AWS Batch

[limnh]

What is the agenda:

• Ashley's out on vacation for two weeks.
• I'll be working to assess AWS Batch and its use case for Simon.
• Tasks to be completed by time Ashley returns:
  • GDoc living document for findings/questions/how does this solve our problem?
  • An outline for an RFC

Daily To-Do List:

Issues/Questions:

• Spot Batch jobs? Is this a way to separate out low-priority jobs and wait for resources to be at a certain monetary threshhold before running? Would this help larger more critical jobs from failing?

Thoughts/Solutions:

• As much as I hate watching technical tutorials where folks just talk at you, this tutorial was surprisingly helpful.
  • Curiosity about how spot instances are utilized/if they can be used at Simon.
• Found the Terraform provider resource for AWS Batch creation
  • Won't probably get to today, but worth calling out for future nicole.

Glossary of notable terms (taken from here):

• AWS Batch: a regional service that simplifies running batch jobs across multiple Availability Zones within a region. You can create AWS Batch compute environments within a new or existing VPC. After a compute environment is up and associated with a job queue, you can define job definitions that specify which Docker container images to run your jobs. Container images are stored in and pulled from container registries, which may exist within or outside of your AWS infrastructure.

• Jobs: A unit of work (such as a shell script, a Linux executable, or a Docker container image) that you submit to AWS Batch. It has a name, and runs as a containerized application on an Amazon EC2 instance in your compute environment, using parameters that you specify in a job definition. Jobs can reference other jobs by name or by ID, and can be dependent on the successful completion of other jobs.

• Job Definitions: A job definition specifies how jobs are to be run; you can think of it as a blueprint for the resources in your job. You can supply your job with an IAM role to provide programmatic access to other AWS resources, and you specify both memory and CPU requirements. The job definition can also control container properties, environment variables, and mount points for persistent storage. Many of the specifications in a job definition can be overridden by specifying new values when submitting individual Jobs.

• Job Queues: When you submit an AWS Batch job, you submit it to a particular job queue, where it resides until it is scheduled onto a compute environment. You associate one or more compute environments with a job queue, and you can assign priority values for these compute environments and even across job queues themselves. For example, you could have a high priority queue that you submit time-sensitive jobs to, and a low priority queue for jobs that can run anytime when compute resources are cheaper.

• Compute Environment: A compute environment is a set of managed or unmanaged compute resources that are used to run jobs. Managed compute environments allow you to specify desired instance types at several levels of detail. You can set up compute environments that use a particular type of instance, a particular model such as c4.2xlarge or m4.10xlarge, or simply specify that you want to use the newest instance types. You can also specify the minimum, desired, and maximum number of vCPUs for the environment, along with the amount you are willing to pay for a Spot Instance as a percentage of the On-Demand Instance price and a target set of VPC subnets. AWS Batch will efficiently launch, manage, and terminate EC2 instances as needed. You can also manage your own compute environments. In this case you are responsible for setting up and scaling the instances in an Amazon ECS cluster that AWS Batch creates for you.

[limnh]

Seriously, do we use EC2 spot instances anywhere? Could this be a possible way forward for OKRs of not spending so much damn money?