We have occasionally run into trouble with batch compute instances running out of disk space during nextflow runs. We tried to solve this by making two queues, one with larger drives be default, but this is annoying to track. It looks like there is a way to set up an instance to autoexpand a volume with a daemon that checks space and allocates more space when nearing capacity.
It all looks fairly straightforward, but with my limited experience with AWS, it might be more efficient for someone like @davidsmejia or @arkid15r to have a look?
I think the plan would be to set up a separate batch queue/compute environment based on an AMI or launch template with the EBS autoscaler installed, then we can test that with some nextflow workflows.
We have occasionally run into trouble with batch compute instances running out of disk space during nextflow runs. We tried to solve this by making two queues, one with larger drives be default, but this is annoying to track. It looks like there is a way to set up an instance to autoexpand a volume with a daemon that checks space and allocates more space when nearing capacity.
Some docs for this are here: https://docs.opendata.aws/genomics-workflows/core-env/create-custom-compute-resources.html and the repo for the autoscale daemon is https://github.com/awslabs/amazon-ebs-autoscale
It all looks fairly straightforward, but with my limited experience with AWS, it might be more efficient for someone like @davidsmejia or @arkid15r to have a look?
I think the plan would be to set up a separate batch queue/compute environment based on an AMI or launch template with the EBS autoscaler installed, then we can test that with some nextflow workflows.