AAE feature request: provide AAE hash tree build/exchange IO limits

[deanproctor]

In testing with large vnodes (> 1TB/vnode) it was found that a single AAE tree build could consume more than 200MB/s of disk IO. A single hash tree exchange could generate up to 800MB/s of network IO. Tree builds typically took 45 minutes to 2.5 hours to complete. It is unclear how long each tree exchange took to complete. When under load, the impact of AAE was to reduce cluster throughput by roughly 33%.

Two solutions are suggested:

• Short term solution - provide the ability for the administrator to define windows during which AAE builds and exchanges are allowed to run, similar to Bitcask merge windows. This would allow the administrator to constrain the impact of AAE (other than async tree writes) to off-peak periods.
• Long term solution - provide the ability for the administrator to cap AAE builds and exchanges to specific disk and network IO limits.

Reference:

Basho Bench log showing AAE on vs AAE off: https://gist.github.com/deanproctor/641ca8797b4ffd4eac90

Cluster specs:

• 6 nodes
• 256 ring size
• Riak CS mutli-backend
• anti_entropy_concurrency = 1
• AAE leveldb defaults

Node specs:

• Supermicro model 6017R-72RFTP
• 1U Chassis
• Dual Intel Xeon E5-2620 CPUs @ 2GHz with Hyperthreading (24 logical cores)
• 128GB 1333MHz ECC memory
• Dual 10GbE Broadcom NetXtreme II BCM57810 network interfaces
• Dual SAS2008 PCI-Express Fusion-MPT SAS-2 Controller
• 84 Western Digital (WD4001FYYG-01SL3) 4TB 7200RPM 6GB/s SAS drives

[jtuple]

This was handled in #765