ubarsc / rios

A raster processing layer on top of GDAL
https://www.rioshome.org
GNU General Public License v3.0
15 stars 8 forks source link

Various improvements to AWS Batch support #67

Closed gillins closed 1 year ago

gillins commented 1 year ago

This PR adds the following:

In future I suspect it may be necessary to ensure that the "main" script has access to more local storage as you'd only be using this if you had very large files and by default Batch jobs don't have much local storage. This can be done by either adding a LaunchTemplate to the ComputeEnvironment that increases the size of the root volume, or my directly creating and mounting an EBS Volume. RIOS should probably do something here to make it easy, but I'll make that a separate PR.

neilflood commented 1 year ago

That seems all very good (as if I understood it....). My only quibble is to request that you change the name of the region environment variable. Currently you have it as RIOS_BATCH_REGION, but I would prefer RIOS_AWSBATCH_REGION, just to be completely clear that this relates only to AWS Batch.

Sometimes my nit-picking amazes even me........

gillins commented 1 year ago

Good thought!

neilflood commented 1 year ago

I just bothered to check, and it appears you already had a RIOS_BATCH_STACK variable as well. Sorry, I had not noticed that. Anyway they should be consistent, so I guess that means change that to RIOS_AWSBATCH_STACK. Does it???

Sorry, I am not being a very good reviewer.

gillins commented 1 year ago

I'd forgotten about that - good spotting. PR's are great aren't they.