Implement the MVP for distributed processing - Githubissues

lakehq / sail

LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

https://lakesail.com

Apache License 2.0

369 stars 11 forks source link

Implement the MVP for distributed processing #246

Open linhr opened 3 weeks ago

linhr commented 3 weeks ago

User Interface

[ ] Sail CLI (#245)
[ ] Sail configuration

Core Functionalities

[x] Distributed processing setup (#244)
[x] Distributed job stages and shuffle (#265)
[ ] Worker scaling and reuse

Resource Provider Integration

[ ] Kubernetes resource provider
[ ] YARN resource provider

Observability

[ ] OpenTelemetry setup for driver/worker nodes

shehabgamin commented 5 days ago

There are these PRs as well:

Implement actor system (#264).
Improve actor error handling (#263).
Improve actor implementation (#262).
Implement physical plan extension codec (#259).