search

apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine
https://datafusion.apache.org/ballista
Apache License 2.0
1.53k stars 194 forks source link

Remove redundant fields in ExecutorManager #728

Closed yahoNanJing closed 1 year ago

yahoNanJing commented 1 year ago

Which issue does this PR close?

Closes #723.

Rationale for this change

After the cluster state refactored by #658, the state cache part in ExecutorManager becomes redundant. It's better to remove them.

What changes are included in this PR?

  1. The following fields in ExecutorManager are removed:

    • slots_policy
    • executor_metadata
    • executors_heartbeat
    • executor_data

  2. And the RoundRobinLocal slot policy is removed.

  3. And the SlotsPolicy is renamed to TaskDistribution.

  4. Removed buggy executor check for pull-staged task scheduling

Are there any user-facing changes?

yahoNanJing commented 1 year ago

Hi @thinkharderdev, could you help review this PR?

yahoNanJing commented 1 year ago

Hi @thinkharderdev, now I copy the cache logic into the KeyValueState, could you help have a check?