apache / accumulo

Apache Accumulo
https://accumulo.apache.org
Apache License 2.0
1.06k stars 445 forks source link

Collapse Compaction coordinator code into the Manager #4220

Open ddanielr opened 7 months ago

ddanielr commented 7 months ago

In elasticity, the coordinator has been collapsed into the manager to reduce complexity around running number of different accumulo servers.

As we have standardized on using external compactions and deprecated internal compactions in 3.1, we should also migrate the coordinator code into the manager.

This should also replace the top level compaction entry from the cluster.yaml with compactor

compactor:
    - q1:
        - localhost1
        - localhost2
    - q2:
        - localhost3
        - localhost4
keith-turner commented 7 months ago

This is a nice change because it makes the 3.1 release have the following

This avoids encouraging use of external compactions in 3.1 and changing the user experience for that feature in 4.0

ddanielr commented 7 months ago

This is a nice change because it makes the 3.1 release have the following

  • removes the experimental status of external compaction
  • external compactions in 3.1 and 4.0 have a similar user experience

This avoids encouraging use of external compactions in 3.1 and changing the user experience for that feature in 4.0

To expand on the user experience part:

In 2.1 we expect the user to configure a new server type of "compaction-coordinator" to use external compactions. This requires a modification to the cluster.yaml but is considered an experimental configuration change.

In 3.1 we change the feature to be supported and there are no direct user changes.

In elasticity we drastically change the cluster.yaml format and remove the concept of a separate compaction-coordinator server.

By moving the compaction-coordinator change down into 3.1, we reduce a lot of the configuration churn for users and make it part of the expected step of a feature moving from experimental status to supported.

keith-turner commented 7 months ago

While working on #4225 I found that MiniAccumulo has a compaction coordinator server type in its public API. That will need to be deprecated and its use made to be a noop.

dlmarion commented 4 months ago

In elasticity, I moved the Coordinator into the Manager in PR #3216. Someone could use this as a guide to account for all of the things that need to be changed.