Live reloading configs

[fsamuel-bs]

AtlasDB, both client and timelock-server, should support live reloaded configs - this is, changes on configs after the service has started should be reflected on the behavior of the application.

[hsaraogi]

Partially fixed by #1976.

[gsheasby]

This could be finished off early next year by adding live reloading for KVS config.

Comment from @tboam: "The KVS config which includes the list of C* servers is not currently live reloaded. I actually think the Cassandra Client Pools code supports this and we just need to move it to the runtime config and wire it up. Will prioritise for very early next year. We need to figure out if there's anything that can't be live reloaded."

[hsaraogi]

Ensure all Atlas config is live-reloadable or have a good reason for why some blocks arent.

[gsheasby]

The following sections of the config are not live reloaded:

• KeyValueServiceConfig (partially covered by #2839)
• Leader, Lock and Timestamp blocks - all three are obsolete
• TimeLockClientConfig - partly done in #1771 ? The remainder is high lift, and we won't tackle it soon
• Various individual variables:
  • Five relating to Scrub (should be bundled into a live-reloadable ScrubConfig - would INTERNAL_LARGE_PRODUCT care about this?)
  • Eight deprecated configs relating to sweep
  • initialiseAsync (no purpose for live reloading this)
  • namespace (should not normally be changed)
  • transactionReadTimeoutMillis
  • punchIntervalMillis
  • kvsSlowLogThresholdMillis
  • defaultLockTimeoutSeconds

So the remaining work here is to move the four individual variables to AtlasDbRuntimeConfig, and to create a live-reloadable ScrubConfig block (assuming these variables would actually be used by someone).