search

digitalearthpacific / dep-tools

Processing tools for Digital Earth Pacific
MIT License
1 stars 0 forks source link

Add load-before-write option #43

Closed alexgleith closed 8 months ago

alexgleith commented 8 months ago

I was getting a few failures before, something to do with Dask aborting.

Adding this 'load-before-write' flag means that it'll process everything into memory before trying to write files, and seems to fix the issue.

jessjaco commented 8 months ago

I think I'd prefer you did this in your processor. Putting it here limits flexibility (by adding more params but also assuming we're processing xarray objects)

alexgleith commented 8 months ago

Yeah, ok, good point.

jessjaco commented 7 months ago

FYI I'm putting this back in, but in a different place (in the writer after optional data type changes back to int). See #44