search

ubarsc / rios

A raster processing layer on top of GDAL
https://www.rioshome.org
GNU General Public License v3.0
15 stars 8 forks source link

Split main and worker queues. Allow storage to be overridden #112

Closed gillins closed 2 months ago

gillins commented 3 months ago

The main thing this PR adds the ability to increase the storage available for the 'main' worker. If creating large files this will be very useful as by default the local storage on a Batch worker is quite small.

To achieve this I needed to split the main and worker jobs into different ComputeEnvironments and thus different queues which may assist keeping everything separate.

I'm also considering allowing the placement group strategy to be overridden, but haven't done so yet. Please don't merge this until I've done some more testing.

neilflood commented 2 months ago

Wow. It only gets more complicated :-). Well done, though, especially the notes on setting up in their own VPC.

A couple of typos:

Very good.