This PR does three main things in service of making our AWS batch settings more flexible for future use (and allowing running cellranger in the near future).
Closes #14
First, it switches the AMI used to one based on Amazon Linux 2, which seems to make use of a more flexible docker setup with regard to the space available to the docker image.
Second, it uses Launch Templates to allow the same AMI to be used to launch instances with varying disk sizes. There are two templates currently implemented: one with a 64GB EBS volume, the other with a 500GB volume.
Third, these Launch templates are now used to create the compute environments and batch queues. I have replaced the "priority" queue with a "bigdisk" queue.
I have tested these changes by submitting previous jobs to the default spot queue, and it seems to work! I have not explicitly checked the bigdisk queue, but I see no reason it should not also work as expected. It will be tested with the cellranger workflow that is coming soon, and I expect any changes required to be implemented at that point.
jashapiro
commented
4 years ago
This PR does three main things in service of making our AWS batch settings more flexible for future use (and allowing running cellranger in the near future).
Closes #14
First, it switches the AMI used to one based on Amazon Linux 2, which seems to make use of a more flexible docker setup with regard to the space available to the docker image.
Second, it uses Launch Templates to allow the same AMI to be used to launch instances with varying disk sizes. There are two templates currently implemented: one with a 64GB EBS volume, the other with a 500GB volume.
Third, these Launch templates are now used to create the compute environments and batch queues. I have replaced the "priority" queue with a "bigdisk" queue.
I have tested these changes by submitting previous jobs to the default spot queue, and it seems to work! I have not explicitly checked the
bigdiskqueue, but I see no reason it should not also work as expected. It will be tested with thecellrangerworkflow that is coming soon, and I expect any changes required to be implemented at that point.