Alluxio / alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud
https://www.alluxio.io
Apache License 2.0
6.86k stars 2.94k forks source link

Add cluster usage balance CLI #15517

Open JySongWithZhangCe opened 2 years ago

JySongWithZhangCe commented 2 years ago

Is your feature request related to a problem? Please describe. I'm always frustrated when few Alluxio workers are full of data, while others are empty.

Describe the solution you'd like Add cluster usage balance CLI to manually balance the cache.

Describe alternatives you've considered

Urgency Urgent

Additional context Nope.

JySongWithZhangCe commented 2 years ago

@jiacheliu3 Would you like to give some suggestions?

jiacheliu3 commented 2 years ago

fyi @HelloHorizon

JySongWithZhangCe commented 2 years ago

For now, the applications has final and absolute control over how the cached data is distributed. In this case, excellent users can speed up data analysis well, while unprofessional users encounter various problems. When Alluxio provides services as a sharing platform, it is inevitable that unprofessional users will affect the use of other users.

At present, the administrator does not have a good way to intervene, and can only solve the problem by stopping the worker, deleting the cache, and starting the worker. Administrators need more tools to manage specific workers.

JySongWithZhangCe commented 2 years ago

The uneven distribution of data directly leads to the skew of task scheduling in analysis engines such as Spark, and hotspot workers are frequently under pressure from cache replacement.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.