search

pangeo-data / rechunker

Disk-to-disk chunk transformation for chunked arrays.
https://rechunker.readthedocs.io/
MIT License
162 stars 25 forks source link

Refactor executors again #112

Closed rabernat closed 2 years ago

rabernat commented 2 years ago

This was a little coding project while stuck in a plane for a few hours. Here I am basically just copying over pangeo-forge-recipes' latest executors, which we spent a lot of time optimizing.

Locally I have pywren tests failing with a message TypeError: cannot pickle 'weakref' object. Was not able to debug this quickly as I'm not too familiar with pywren.

tomwhite commented 2 years ago

Locally I have pywren tests failing

There has not been a pywren release for over 3 years, since it has now become lithops. I wonder if we should remove or disable the pywren executor for this reason.

codecov[bot] commented 2 years ago

Codecov Report

Merging #112 (ac9737f) into master (0a0d1eb) will decrease coverage by 0.48%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #112      +/-   ##
==========================================
- Coverage   96.81%   96.32%   -0.49%     
==========================================
  Files          12       11       -1     
  Lines         534      490      -44     
  Branches      121      112       -9     
==========================================
- Hits          517      472      -45     
  Misses         11       11              
- Partials        6        7       +1
Impacted Files Coverage Δ
rechunker/executors/dask.py 100.00% <100.00%> (+5.45%) :arrow_up:
rechunker/executors/prefect.py 100.00% <100.00%> (ø)
rechunker/executors/python.py 100.00% <100.00%> (ø)
rechunker/pipeline.py 100.00% <100.00%> (ø)
rechunker/types.py 100.00% <100.00%> (ø)
rechunker/api.py 98.04% <0.00%> (-1.96%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 0a0d1eb...ac9737f. Read the comment docs.

rabernat commented 2 years ago

Ok so in fb8b1ee I just removed the pywren executor. If it is unmaintained, it's not feasible for us to find workarounds to keep supporting it. @tomwhite - are you using the pywren executor these days?

I had a look at lithops, and it looks like it should be pretty straightforward to support. It would be nice to have a serverless option. If someone wants to implement and maintain a lithops executor, I would be happy to support that effort.

tomwhite commented 2 years ago

@tomwhite - are you using the pywren executor these days?

No, I'm not.

rabernat commented 2 years ago

For some reason the test workflow is not running.

TomAugspurger commented 2 years ago

GitHub actions was having issues earlier, but nothing is reported at https://www.githubstatus.com/ at the moment.

TomAugspurger commented 2 years ago

Oh, https://github.com/pangeo-data/rechunker/actions/runs/2029808478 shows

The workflow is not valid. .github/workflows/ci.yaml (Line: 9, Col: 7): Unexpected value 'fast-fail'