Implement the MVP for distributed processing - Githubissues

lakehq / sail

LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

https://lakesail.com

Apache License 2.0

525 stars 13 forks source link

Implement the MVP for distributed processing #246

Open linhr opened 1 month ago

linhr commented 1 month ago

User Interface

[x] Sail CLI (#245)
[x] Sail configuration (#279)

Core Functionalities

[x] Distributed processing setup (#244)
[x] Distributed job stages and shuffle (#265)
[x] Worker scaling and reuse (part of #280)

Resource Provider Integration

[x] Kubernetes resource provider (#280)
[ ] YARN resource provider

Observability

[ ] OpenTelemetry setup for driver/worker nodes

shehabgamin commented 3 weeks ago

There are these PRs as well:

Implement actor system (#264).
Improve actor error handling (#263).
Improve actor implementation (#262).
Implement physical plan extension codec (#259).

shehabgamin commented 1 week ago