apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
5.5k stars 1.02k forks source link

[Ballista] Improve task and job metadata #472

Open edrevo opened 3 years ago

edrevo commented 3 years ago

The task and job status we save in the scheduler state is currently lacking. See:

https://github.com/apache/arrow-datafusion/blob/16011120a1b73798049c5be49f9548b00f8a0a00/ballista/rust/core/proto/ballista.proto#L669-L680

And

https://github.com/apache/arrow-datafusion/blob/16011120a1b73798049c5be49f9548b00f8a0a00/ballista/rust/core/proto/ballista.proto#L614-L624

The job status should include:

The task status should include:

edrevo commented 3 years ago

cc @pradomota. I'm opening this one in case you want to take a stab at it 😉 We can do pair programming if you want.

msathis commented 3 years ago

This will be great addition. We can expose this information to the UI as well.