Improve graceful shutdown of RegionSevers

[sbernauer]

Relevant docs: https://hbase.apache.org/book.html#decommission Relevant script: graceful_stop.sh Relevant class: org.apache.hadoop.hbase.util.RegionMover, with relevant function

In https://github.com/stackabletech/hbase-operator/issues/400 we implemented a graceful shutdown for all HBase components which is similar to ./bin/hbase-daemon.sh stop <service>. While this works in general it has downsides, such regions being offline for some time, resulting in (short) outages.

Instead we should try to call or mimic graceful_stop.sh. The graceful_stop.sh script will move the regions off the decommissioned RegionServer one at a time to minimize region churn. It will verify the region deployed in the new location before it will moves the next region and so on until the decommissioned server is carrying zero regions. At this point, the graceful_stop.sh tells the RegionServer stop. The master will at this point notice the RegionServer gone but all regions will have already been redeployed and because the RegionServer went down cleanly, there will be no WAL logs to split.

### Acceptance criteria
- [ ] Must: Call or mimic `graceful_stop.sh`
- [ ] Must: The docs say "Disable the Load Balancer before Decommissioning a node". We found a solution to this by either doing so or making sure we (or our customers) are not using LBs
- [ ] Should: Decommissioning several Regions Servers concurrently: To gracefully drain multiple regionservers at the same time, RegionServers can be put into a "draining" state. This is done by marking a RegionServer as a draining node by creating an entry in ZooKeeper under the hbase_root/draining znode. Watch out to clean up or make sure the regionserver does this when starting up again

[NickLarsenNZ]

Must: The docs say "Disable the Load Balancer before Decommissioning a node". We found a solution to this by either doing so or making sure we (or our customers) are not using LBs

Can we just use readiness probes to take the pod out of service?

[razvan]

There need to be at lease two shutdown modes:

• one where regions are being moved around because the service is decommissioned forever. This one is slow and possibly generates a lot of traffic inside the cluster.
• a fast one temporary decommissioned servers due to security, version updates and what not. The region balancer should probably be stopped during the entire time.

Findings (in progress):

• hbase/bin/hbase-daemon.sh can start/stop/restart etc. and already handles termination signals better then our home grown solution.
  • uses jstack to do a thread dump in case shutdown takes longer than 20 mins. jstack is not in our images.
• The graceful_stop.sh script requires the hostname or ssh commands which are not available in the Hbase images currently.
  • It always moves regions to a different server
  • It can turn off region balacing before shutdown and turn it on again when a server is stopped.
  • We can get rid of the ssh requirement by passing localhost as the name of the region server but the script needs hostname to find out the actual region server name.
  • Assumes the HBase servers have been started with hbase-daemon.sh which writes PID files for every process.
• An additional way to decommission a region server is the decommission_regionserver shell command which also can move regions (async) but doesn't actually stop anything. A mechanism to wait for the regions to be moved is needed in this case.

[razvan]

During testing it was discovered that region servers already transfer regions when shutting down. This behavior is implemented in the 2.4 and 2.6 versions.

To clarify:

• What is the benefit of invoking the region mover explicitly before shutdown?
• Are regions in "transition" available for querying ?
• How long can a region move take in the worst case and how does this impact HBase clients ?

Another idea : since this is the default behavior anyway, maybe in cases like rolling cluster restarts, the user would benefit more from actually disabling the region mover altogether during that period.

[NickLarsenNZ]

This will be discussed next week