search

Alluxio / alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud
https://www.alluxio.io
Apache License 2.0
6.84k stars 2.94k forks source link

Decouple Allocator and Reviewer #13289

Open jiacheliu3 opened 3 years ago

jiacheliu3 commented 3 years ago

Summary Now the Reviewer instance is in the Allocator instance, eg.https://github.com/Alluxio/alluxio/blob/5970957e176fccc93e3fd5cffe15ee59b8ba178d/core/server/worker/src/main/java/alluxio/worker/block/allocator/RoundRobinAllocator.java#L57 This design is wrong. The coupling should be removed.

The main challenge of the decouple is the fact that we have both stateless(MaxFreeAllocator which simply gives you the most free dir) and stateful(RoundRobinAllocator). It's hard to use the same design and at the same time change as little of the interface as possible. For the stateful one the main challenge is how the state should be maintained when the decision can be rejected and then you retry. For the stateless one, it's hard to reject and ask for a different decision.

After a discussion with @ggezer , here are the thoughts:

From the design perspective, Allocator and Reviewer should still be decoupled. We will need to examine over each Allocator+Reviewer combination to see if that makes sense. Based on (2) we can come up with the overseer design(currently AllocationCoordinator in this PR but that can be a better name). That design should make sense for the combinations we examine and therefore the implementation should be able to last longer.

Urgency MEDIUM

It'll be good if we resolve this before 2.6.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

elega commented 1 year ago

@jiacheliu3 is this issue still valid?

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.