Stored application settings (Experimental)

Problem statement

Currently, application settings are obtained from static Json text files, usually named appsettings.json or appsettings.[environment].json.

This is configured at the start of the application, like this:

    config.AddJsonFile("configs/appsettings.json", false, true);
    config.AddJsonFile($"configs/appsettings.{env}.json", false, true);

The option reloadOnChange is true in the above code, which means you can use the options pattern with the IOptionsSnapshot<> interface to read the settings value which will reflect the current value in the file. If the value in the file is updated, the updated value will be read.

For example, suppose you have the following definition:

public class DemoSettings
{
    public string? Value1 { get; set; }
    public int? Value2 { get; set; }
    public string[]? Value3 { get; set; }
}

In your appsettings.json file, you have:

  "MySettings": {
    "Value1": "Hello World",
    "Value2": 42,
    "Value3": [ "1", "2", "3" ]
  }

... and in your Program.cs you define:

    services.Configure<DemoSettings>(Configuration.GetSection("MySettings"));

The resulting configuration can be read in some scoped service as follows:

[Export(typeof(ISomeService)), Scoped]
public class SomeService : ISomeService
{
    private readonly DemoSettings _settings;

    public SomeService(IOptionsSnapshot<DemoSettings> options)
    {
        _settings = options.Value;
    }

What is put in appsettings.json is not expected to change (often).

Application configuration which is subject to (regular) changes should really be obtained from some persistent store like a database, and are normally not part of appsettings.json. The decision to store options in the appsettings.json file or somewhere else is a business decision.

However, updating appsettings.json means re-deploying the application or building a special deployment. This is not ideal.

It would be better if we had some kind of mechanism to read settings from a persistent store (like a database). This way, the initial values would come from our appsettings.json but the database would allow these values to be updated and propagated to our application.

This pull request is such a mechanism.

Adding the capability

Two new DLLs have been defined:

Arc4u.Standard.Configuration.Store contains the mechanism to specify the storage of configuration settings, without actually defining the precise kind of persistent store. This is compatible with .NET Standard 2.x and .NET Core 6, 7 and 8.
Arc4u.Configuration.Store.EFCore contains a persistent store defined using Entity Framework Core (EF Core). This is compatible with .NET Core 6, 7 and 8. (since EF Core isn't compatible with .NET Standard).

Startup code additions

In your Program.cs, you need to specify which section(s) of your application settintgs need to be persisted to storage. To reprise the previous example, you would write:

    config.AddJsonFile("configs/appsettings.json", false, true);
    config.AddJsonFile($"configs/appsettings.{env}.json", false, true);
    config.AddSectionStoreConfiguration(options =>
    {
        options.Add<DemoSettings>("MySettings");
    });

You just specify which sections you want. There is no need to change the Json file.

You can also specify an instance explicitly. For example, suppose you need to define another section called "MyOtherSettings" and you don't want to modify your appSettings.json, you can just add the instance directly:

    config.AddSectionStoreConfiguration(options =>
    {
        options
        .Add<DemoSettings>("MySettings")
        .Add("MyOtherSettings", new DemoSettings { Value1 = "V1", Value2 = 43 });
    });

You configure your settings by calling services.Configure<> as usual.

Let's assume from this point you want to add the configuration settings to your existing database. You probably have something like this:

    services.AddDbContextFactory<DatabaseContext>(optionsBuilder => ...);

After the above AddDbContextFactory or AddDbContext, you add these two lines:

    services.AddDbContextSectionStore<DatabaseContext>();
    services.AddSectionStoreService();

Here, DatabaseContext is the name of your database context.

Finally, after your app is built, you need to add:

   app.Services.UseSectionStoreConfiguration();

Data layer additions (EF Core)

In the definition of your DatabaseContext, you need to include the entity for persisting the sections. The type is called SectionEntity.

There is a builder to configure that entity correctly.

This happens in the model builder. For relational models, you need to add the following code in OnModelCreating:

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        // any existing code here...

        // Configure the model for a SectionEntity table
        modelBuilder.Entity<SectionEntity>()
            .Configure()
            .ToTable("AppSettings");
    }

The name (and the schema) of the table can be freely chosen.

You don't have to use your existing database context. You can define a separate one if you wish, for the sole purpose of holding your sections.

Behavior

When the application is building (i.e. before app.Build()), the values of the sections will be the default values as configured in the call to AddSectionStoreConfiguration.

It is only after the application is built that any new values are read from persistent storage.

The first time the application starts, the persistent store will be populated with the default values as specified in your appsettings.json or expicitly. After the call to app.Services.UseSectionStoreConfiguration() the values will be read from the persistent store.

Modifying section values

Constructing the user interface and doing validation for modifying configuration section values is up to you.

There is an interface called ISectionStore defined as follows:

/// <summary>
/// A contract to (re)read or construct the collection of <see cref="SectionEntity"/>s from persistent storage
/// </summary>
public interface ISectionStore
{
    /// <summary>
    /// Get the contents of a specific section
    /// </summary>
    /// <param name="key"></param>
    /// <param name="cancellationToken"></param>
    /// <returns>the specific <see cref="SectionEntity"/> or null if there is no such section.</returns>
    Task<SectionEntity?> GetAsync(string key, CancellationToken cancellationToken);

    /// <summary>
    /// Get all sections in the database for a specific version
    /// </summary>
    /// <returns>The list of all section entities. Modifying this list has no effect on the store</returns>
    List<SectionEntity> GetAll();

    /// <summary>
    /// Update one or more section entities in the store
    /// </summary>
    /// <param name="entities"></param>
    /// <param name="cancellationToken"></param>
    /// <returns></returns>
    Task UpdateAsync(IEnumerable<SectionEntity> entities, CancellationToken cancellationToken);

    /// <summary>
    /// Reset the store (i.e. remove all the <see cref="SectionEntity"/> records, causing the app to revert to the initial data
    /// </summary>
    /// <param name="cancellationToken"></param>
    /// <returns></returns>
    Task ResetAsync(CancellationToken cancellationToken);

    #region Internal use

    /// <summary>
    /// This is called to add the initial set of section entities. It is not intended to be called from user code.
    /// </summary>
    /// <param name="entities"></param>
    void Add(IEnumerable<SectionEntity> entities);

    #endregion
}

This is implemented using a scoped service, which can be injected:

[Export(typeof(ISomeService)), Scoped]
public class SomeService : ISomeService
{
    private readonly ISectionStore _store;

    public SomeService(ISectionStore store)
    {
        _store = store;
    }

After that, you can update some property of a section simply. For example, this is a piece of code that modifies MySettings:Value2:

    private async Task ModifyValue2(int newValue2, CancellationToken cancellation)
    {
        var entity = await _store.GetAsync("MySettings", cancellation).ConfigureAwait(false);
        if (entity is not null)
        {
            var settings = entity.GetValue<DemoSettings>();
            if (settings is not null)
            {
                settings!.Value2 = newValue2;
                entity.SetValue(settings);  // important! 
                await _store.UpdateAsync(entity, cancellation).ConfigureAwait(false);
            }
        }
    }

We get the Value2 before overwriting it, to illustrate how to get a value, even though the original value is never used.

The important thing to note here is that the SectionEntity is just a wrapper for a key and a value. The Key is immutable and is determined by the default values during configuration. The value is manipulated using GetValue<> and SetValue<>.

This allows us to make a distinction between a null SectionEntity, which means that there is no such section (which is abnormal and should not happen), and a null Value, which means that the value is null (which may be legal, depending on the application).

Because you will most likely deal with values directly, there are extension methods on the ISectionStore interface to ease the pain:

public static async Task<(bool Valid, TValue? Value)> TryGetValueAsync<TValue>(this ISectionStore sectionStore, string key, CancellationToken cancellationToken);

public static async Task<bool> TrySetValueAsync<TValue>(this ISectionStore sectionStore, string key, TValue? value, CancellationToken cancellationToken);

With those, the above code becomes:

    private async Task ModifyValue2(int newValue2, CancellationToken cancellation)
    {
        var success = await _store.TrySetValue<DemoSettings>("MySettings", newValue2, cancellation).ConfigureAwait(false);
        // success is true if the value was set successfully
     }

The ISectionStore.Add method is used to populate the existing entries and should never be used by user code.

The ISectionStore.ResetAsync removes all changes from the database and rebuilds it with the default values.

Issues

Distributed services

Because there can be many deployments of a single service, there is a monitoring mechanism that periodically polls the database and notifies the configuration provider if something has changed. There can be therefore a delay between updating a section value and "seeing" it in the services.

By default, this polling happens every 15 seconds. This can be adjusted by passing a TimeSpan in the call to AddSectionStoreService.

Versioning

There is no good mechanism for versioning section values. You are advised to add a version number to their section types if they think this is an issue. Another alternative would be to add updated sections with new (distinct) keys.

GFlisch / Arc4u