Closed mrocklin closed 9 months ago
It might also be fun to look at optimizing costs with arm/spot instance sizing and see what we can get this down to.
I happened to already be working on this and have most of the changes needed in a notebook locally. I'll push that up into a PR here shortly
This example is great at showcasing embarassingly parallel workflows. It predates the coiled.function decorator, but we should probably update it to use that instead of futures.
@nathanballou want to take a crack at this?
One challenge will be that there's the single file (I think it's a directory?) where the we run into a failure. We handle this in the current version by skipping all erred futures. We'll have to handle that differently here. Some options:
exceptions="skip"
)This is generally a good first issue, but maybe less good if we use it on Thursday during a demo (I may want to use this then). I mention this because I reserve the right come in here and do this if it's taking a while.