coiled / examples

Examples using Dask and Coiled
14 stars 3 forks source link

Update arXiv Matplotlib example to use Coiled functions #33

Closed mrocklin closed 9 months ago

mrocklin commented 9 months ago

This example is great at showcasing embarassingly parallel workflows. It predates the coiled.function decorator, but we should probably update it to use that instead of futures.

@nathanballou want to take a crack at this?

One challenge will be that there's the single file (I think it's a directory?) where the we run into a failure. We handle this in the current version by skipping all erred futures. We'll have to handle that differently here. Some options:

  1. Figure out what the directory is and remove it manually from the list of files
  2. Add some exception handling to coiled functions themselves (exceptions="skip")
  3. Handle it in the loop when we iterate over things (but maybe the batching will make this awkward)

This is generally a good first issue, but maybe less good if we use it on Thursday during a demo (I may want to use this then). I mention this because I reserve the right come in here and do this if it's taking a while.

mrocklin commented 9 months ago

https://github.com/coiled/examples/blob/main/arxiv-matplotlib.ipynb

mrocklin commented 9 months ago

It might also be fun to look at optimizing costs with arm/spot instance sizing and see what we can get this down to.

jrbourbeau commented 9 months ago

I happened to already be working on this and have most of the changes needed in a notebook locally. I'll push that up into a PR here shortly