Wukong meet Cubed

[TomNicholas]

Hi @Scusemua - I'm raising this because I couldn't find your email anywhere!

@tomwhite and I have been working on Cubed - which is extremely similar to Wukong in it's goals.

If I understand correctly, Wukong aims to create a general-purpose serverless DAG execution framework for data science workloads by building on top of dask.distributed.

Cubed also aims to be a serverless DAG execution framework for data science workloads inspired by Dask, but restricts the problem domain to numpy-like array computations, and does not directly use dask (only borrows some of its API/abstractions). Cubed also uses the cloud-native array storage format Zarr to store state (intermediate arrays) between operations.

Both projects cite PyWren as an inspiration explicitly.

Some of the problems you mention with PyWren are solved by Cubed's approach - in particular on this slide of your talk on Wukong the rapid scaling is handled by serverless frameworks like Lithops, the excessive data movement is handled by writing to Zarr, and the per-function resource limitations are not an issue because each function only needs to process a single chunk.

Of possible interest to you:

• Talk I'm giving tomorrow afternoon on Cubed

• Blog post comparing Cubed and dask.array on a Climate Science Workload

• Pangeo distributed arrays working group, and collection of use cases

[TomNicholas]

Probably also worth mentioning the existence of Coiled Functions, which is also "serverless Dask"

https://docs.coiled.io/user_guide/usage/functions/index.html?utm_source=medium&utm_medium=parallel-coiled-functions

[Scusemua]

Hi Tom!

Thank you for reaching out! 🙂 I will take a look at Cubed as well as Coiled Functions, as I'm not familiar with these. I will also share them with my Wukong colleagues. I'll also attend your talk if I've got time to do so.

If I understand correctly, Wukong aims to create a general-purpose serverless DAG execution framework for data science workloads by building on top of dask.distributed.

Yes, that's correct!

[TomNicholas]

I will take a look

Great!

Talk slides here, and the recording should be posted here soon.

[Scusemua]

Great, sounds good. Thank you. :) I'll check out the slides + recording!