search

azavea / pfb-network-connectivity

PFB Bicycle Network Connectivity
Other
40 stars 10 forks source link

Adjust Batch provisioning for managed compute environment #858

Closed KlaasH closed 7 months ago

KlaasH commented 2 years ago

Issue #857 calls for manually creating a managed Batch compute environment to run the analysis. After we've done that successfully, the next step is to adjust the provisioning/deployment infrastructure to create and use an environment like that rather than the current unmanaged one.

KlaasH commented 1 year ago

AWS has added the option to configure much more ephemeral storage on Fargate tasks (up to 200GB; the old limit was 20GB), so we should be able to skip the managed environment and convert the analysis to Fargate. https://aws.amazon.com/about-aws/whats-new/2023/03/aws-batch-configurable-ephemeral-storage-fargate/

Per https://github.com/azavea/pfb-network-connectivity/issues/839#issuecomment-978232796 (linked from the comment that's linked from the comment above), ~150GB would suffice to provide enough space and a bit of buffer for our biggest jobs.

KlaasH commented 7 months ago

This issue is from an earlier iteration of the plan, where we were talking about moving to a managed EC2-based compute environment. Now we're talking about using Fargate, so this is superseded by issue #941.