search

apache / datafusion-ballista

Apache DataFusion Ballista Distributed Query Engine
https://datafusion.apache.org/ballista
Apache License 2.0
1.4k stars 182 forks source link

Introduce the number of machine cores to sort when executor allocation #837

Closed smallzhongfeng closed 1 year ago

smallzhongfeng commented 1 year ago

Which issue does this PR close?

Closes #832.

Rationale for this change

Better allocation of resources

What changes are included in this PR?

When allocating resources, consider the number of cores of the machine, and at the same time compare it with the number of slots of the corresponding executor, and sort by the ratio.

Are there any user-facing changes?

No.

yahoNanJing commented 1 year ago

I still don't think it's necessary to introduce the machine core for resource allocation. The task slot should be enough.

smallzhongfeng commented 1 year ago

I still don't think it's necessary to introduce the machine core for resource allocation. The task slot should be enough.

Of course, we can continue to discuss, but do you agree with my idea from https://github.com/apache/arrow-ballista/issues/832#issuecomment-1615839482 :-)

smallzhongfeng commented 1 year ago

PTAL if you are free, thanks @thinkharderdev @yahoNanJing