apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine

https://datafusion.apache.org/ballista

Apache License 2.0

1.56k stars 197 forks source link

Fix job hangs when partition count of plan is zero #1024

Closed lewiszlw closed 4 months ago

lewiszlw commented 5 months ago

Which issue does this PR close?

Closes #.

Rationale for this change

When partition count of execution plan is 0, the job will hang because available tasks of running stage will be empty, then get_running_stage_id method of ExecutionGraph will return None. https://github.com/apache/datafusion-ballista/blob/e7cb3d57f61900ea56107c6477c5754429ba821e/ballista/scheduler/src/state/execution_graph.rs#L969 So the running stage will keep running and won't be converted to successful/failed stage.

What changes are included in this PR?

Are there any user-facing changes?

andygrove commented 5 months ago

Thanks @lewiszlw. Could you rebase/upmerge (to fix the CI failure) then I can review