Configuration hot reload

[lucid-at-dream]

So.. It would be great to have a hot reload feature, I'm thinking that we can use this issue to sketch it and eventually implement it.

Intro

So, with this library the idea is that running service or application has a working connection to a Vault that can be used to fetch secrets by key, assuming that the role configured in the program's environment has sufficient permissions within vault.

In the case of this library, we assume that these secrets are meant to be used for configuration purposes. Therefore, it would be a very valuable feature to have the configuration reload automatically when the values of the secrets change in Vault.

Main Concerns

That said, there are two main concerns for me:

1) In the Hosting architecture there are Singleton, Transient and Scoped services. The main difficulty here is with Singleton services, since they're only instantiated once per execution. Since loading the configuration is a typical pattern in constructors, this will likely result in an inability to reload the configuration without restarting the program completely.

2) Will we have to poll vault for new versions of the secrets? Or is there a way of receiving a notification from Vault when there are new secrets?

Configuration Reload

So, as stated above, application that use Singleton services may not be able to successfully reload the configuration without being restarted.

A few ideas:

1) We could achieve a restart by running the application in a child process where the parent process has the business logic necessary to understand whether the child process needs to be killed and re-started.

2) Another possibility, that would work if we assume that there's some external watchdog, would be to simply exit the program altogether.

3) Yet another possibility would be to do nothing and provide the library users with a hook in which they could specify what behavior they would like executed on configuration change.

Now... Having the user's process be spawned as a child of some process that is encoded in this library. I believe that that is extremely intrusive... It would have a high impact on testing, operations, etc. I don't believe it is a viable option.

As for options 2 and 3, I think that we can easily provide both options to the end user. A potential API would be something like:

public static IHostBuilder UseVault(this IHostBuilder builder);

public static IHostBuilder UseVault(this IHostBuilder builder, VaultConfigurationSetup config);

Then we could also apply the builder pattern to VaultConfiguration (perhaps):

VaultConfigurationBuilder
{
  void exitOnConfigurationChange();
  void runCustomHookOnConfiguratioChange(Action<> hook);
}

And the usage would become something like the following:

var vaultConfigSetup = VaultConfigurationBuilder
  .exitOnConfigurationChange()
  .build();

app
  ...
  .useVault(vaultConfigSetup)
  .startAsync();

This is a drafty example, but I believe it would give us both of the poits above (2,3). Since Kubernetes and Docker are becoming ever more popular I think that it is safe to assume that an external watchdog is a very likely scenario and that we can safely assume that shuting down the program and expecting it to be restarted is a viable option.

Listening to Configuration Changes

So... Listening to changes on Vault may cause a quasi DDoS if there are many services trying to reach Vault querying it for changes.

Options here are: 1) Each service polls vault 2) There's a "middle man" that polls vault and provides a subscription mechanism 3) Vault has some native subscription mechanism

Clearly there's still investigation in need of being done here. Namely answering the question of "Does vault have a subscription mechanism for being notified of changes?"

To be continued...

[diogod3]

dotnet already has some methods for hot reload of configurations injected into the services (IOptionsMonitor and IOptionsSnapshot), it is up to the developer to prepare his service for hot reloading (or not).

I think this library should focus only on providing a way for keeping the configurations source updated (and triggering any events that dotnet requires for the previously mentioned methods to work).

This way, we limit the problem to "Listening to Configuration Changes". On this, I don't think Vault has an option to subscribe to changes, so the solution will have to be polling.

The library could offer a HostedService with configurable polling options that would be polling vault and updating configurations when any of them changed.

[Misiu]

What about a solution like this: https://mousavi310.github.io/posts/a-refreshable-sql-server-configuration-provider-for-net-core/#reload-configuration Basically, it creates a timer and reloads the configuration when a configured timespan ends. This works excellently for SqlServer, Redis, and other configuration providers, so it may work fine for Vault.