Allow for configuration changes on the fly

[potuz]

There are several times where a node operator may want to use diffferent CLI flags than the ones that his beacon is currently running. Even if it were for just a few minutes. It would be nice to have a way to signal the beacon that we want to change these flags without the need of a restart. Examples that can be very useful include --disable-monitoring, --enable-debug-rpc-endpoints, --fee-recipient-value --grpc-max-msg-size max-goroutines, and with the upcoming validator monitor --validator-monitor-indices.

[prestonvanloon]

I think Teku does this as well. It might be worthwhile to look at their implementation for an example user flow.

[potuz]

I couldn't find this in their documentation. However I did find that they catch unix signals. For example a kill -HUP teku would trigger a reload of their key directory, this can be used to add validators on the fly. We already do this in prysm, but perhaps the same technique can be adapted to this issue:

• We identify the CLI flags that are succeptible to change without restarting the node
• We trigger a re-read from the configuration yaml in case we catch a HUP

[nisdas]

Changing flags on the fly is tricky for us due to how they are set in all our different services. For some services having it dynamic would make the code a lot lot uglier(ex: p2p).

[rauljordan]

It really depends on what functionality is possible to change dynamically. Some are set on startup and should not be changed, while others are ok to change while the node is running. If we were to proceed with this, we should come up with a list of flags that are safe to change dynamically.