[RFC] Dynamic Configuration Service

[huyaboo]

Proposal

Currently, all configs are statically determined via the config schema defaultValues property and the opensearch_dashboards.yml. This provides a centralized repository of config values but a major limitation is the inability to change configs without a server restart. For instance, if the admin needs to set the server.maxPayloadBytes, they would need to go into the config file, edit it, and restart Dashboards. Additionally, the existing config service does not allow for per-request config values. Say an admin needs to set different CSP rules based on the request. This is not feasible with the current model. The proposal instead is for a new core service dynamicConfigService, an extension of the existing configService, to expose a read-only dynamic config client in the RequestHandlerContext for plugins to consume and obtain configs from an external config store. For core services, the client will be exposed in the dynamicConfigService.start() method.

Background

Currently, the configs are handled by the ConfigService in @osd-config, which takes the defaultValues defined in a plugin/core service config schema and overrides them with the config values found in opensearch_dashboards.yml. Take csp configs for example. Its config schema is the following:

export const config = {  
  path: 'csp',  
  schema: schema.object({  
    rules: schema.arrayOf(schema.string(), {  
      defaultValue: [  
        `script-src 'unsafe-eval' 'self'`,  
        `worker-src blob: 'self'`,  
        `style-src 'unsafe-inline' 'self'`,  
      ],  
    }),  
    strict: schema.boolean({ defaultValue: false }),  
    warnLegacyBrowsers: schema.boolean({ defaultValue: true }),  
  }),  
};

This means that by default, the csp config will be the following:

const DEFAULT_CSP = {
    rules: [`script-src 'unsafe-eval' 'self'`, `worker-src blob: 'self'`, `style-src 'unsafe-inline' 'self'`],
    strict: false,
    warnLegacyBrowsers: true,
};

If we define the following config in the opensearch_dashboards.yml file:

csp.strict: true
csp.rules: ["default-src 'self'"]

The final config exposed to all core services and plugins will be

const DEFAULT_CSP = {
    rules: [`default-src 'self'`],
    strict: true,
    warnLegacyBrowsers: true,
};

For plugins, the config schema can be found in config.ts.

Terminology

As to not introduce confusion later in the doc, the below terms will be used

configService

This is the existing OSD config service, which can be found at @osd-config.

dynamicConfigService

This is an extension of the configService, which enables plugins/core services to define config values in a config store and fetch dynamic values for a subset of configs.

config store

The external store which contains all configurable OSD configs. The default store will be an OpenSearch index .dynamic_opensearch_dashboards

config schema

This is the schema type object used to define the structure, value types, and default values of a core service/plugin config

• For plugins: this is found in the config.ts. Take home plugin for example; its config schema is the following

export const configSchema = schema.object({
    disableWelcomeScreen: schema.boolean({ defaultValue: false }),
    disableNewThemeModal: schema.boolean({ defaultValue: false }),
});

config object

This is a generic object which contains all the values of a given configPath. This is the return value for the dynamicConfigService client getConfig

// Example config object for 'home' plugin
const homePluginConfig = {
    disableWelcomeScreen: false,
    disableNewThemeModal: false,
};

configPath

This is the key value that uniquely identifies configs for all core services and plugins

• For core services: this is part of the path field of the config schema.
  • example: the HTTP config configPath is server, which is found in the config schema
• For plugins: this is the configPath found in the plugin manifest file (opensearch_dashboards.json). If this is not specified in the manifest, the plugin id should be used

async local storage

Async local storage are Node provided classes that enable necessary request context to be exposed throughout a request lifecycle. This is needed to call the dynamicConfigService client since different requests can fetch different config values.

Out of Scope

Creating a management page

To access/update config values from the Dashboards UI, a page should be created. However, the specifics of such a page are outside the scope of this RFC.

Determining which configs can be dynamic

This RFC seeks to establish a framework for setting configs as dynamic. A general guideline for what constitutes a dynamic config is as follows:

• Configs resolved during setup() and cannot be modified are not dynamic
• A config cannot direct affect the plugin dependency graph.
• A plugin cannot be enabled/disabled dynamically. This is because some plugins have hard dependencies on other plugins and functionality assumes the plugin dependency tree is set once. If the enablement can be modified, unexpected behavior can arise.
• OpenSearchService configs cannot be made dynamic as the earliest time the DynamicConfigService can start is after OpenSearchService finishes starting (so that the OpenSearch client can be created)

Plugin owners and core service owners should determine which of their configs can be made dynamic.

High-Level Approach

As stated in the Proposal, there should be a new core service DynamicConfigService, which is an extension of the existing ConfigService. The configs taken from the DynamicConfigService will override the ConfigService configs. In essence, what this means is there is a priority on the config values:

1. If a config value is set in the config store, the value will be used
2. If this value cannot be found in the config store, the value set in the opensearch_dashboards.yml will be used
3. If this value was not set in opensearch_dashboards.yml, the config schema defaultValue will be used

On initialization/before setup()

The DynamicConfigService instance will need to follow ConfigService initialization in order to consume its public methods. It should also be exposed to CoreContext so other core services can consume the service easily.

// This can be found in src/core/server/server.ts

this.logger = this.loggingSystem.asLoggerFactory();  
this.log = this.logger.get('server');  
this.configService = new ConfigService(rawConfigProvider, env, this.logger);  

// Will take configService as a dependency
this.dynamicConfigService = new DynamicConfigService(this.configService, env, this.logger);  

const core = {  
  coreId,  
  configService: this.configService,  
  dynamicConfigService: this.dynamicConfigService,  
  env,  
  logger: this.logger,  
};

In particular, the DynamicConfigService also requires plugins and core services to have their schemas registered by calling setSchema() to leverage existing schema validation logic. While ConfigService already does this, its schema map is private.

// Registering core service config schemas (src/core/server/server.ts)
for (const descriptor of configDescriptors) {  
  if (descriptor.deprecations) {  
    this.configService.addDeprecationProvider(descriptor.path, descriptor.deprecations);  
  }  
  await this.configService.setSchema(descriptor.path, descriptor.schema);  
  this.dynamicConfigService.setSchema(descriptor.path, descriptor.schema);  
}

// Registering plugin config schemas (src/core/server/plugins/plugins_service.ts)
const configDescriptor = plugin.getConfigDescriptor();  
if (configDescriptor) {  
  ...
  await this.coreContext.configService.setSchema(  
    plugin.configPath,  
    configDescriptor.schema  
  );  
  this.coreContext.dynamicConfigService.setSchema(  
    plugin.configPath,  
    configDescriptor.schema  
  );  
}
...

setup()

The DynamicConfigService setup should happen right after the ConfigService sets up. There are several reasons why:

1. ConfigService on setup() will discover all plugin configs and construct a plugin dependency graph. To avoid duplicate efforts, the DynamicConfigService should not concern itself with this
2. ConfigService performs a validation step in its setup() that ensures config schemas are well formed, have unique namespaces, and opensearch_dashboards.yml values are valid and are valid configPaths

When the setup() finishes, the DynamicConfigService will expose the following methods:

export interface InternalDynamicConfigServiceSetup {
    /**
    * Enables other plugins/core services to register their own config store client. By default,
    * an OpenSearch client is provided, but a plugin can register a config store of their choice
    * like PostgreSQL, DDB, MongoDB, etc...
    * 
    */
    registerDynamicConfigClientFactory: (factory: IDynamicConfigStoreClientFactory) => void; 
    /**
    * Enables other plugins/core services to register header keys to be captured by the async local store.
    * For example, registering 'request-id' will make this key accessible to the DynamicConfigClient
    * 
    */
    registerAsyncLocalStoreRequestHeader: (key: string | string[]) => void;
    /**
    * For core services ONLY. This function will return a promise in the case a core service needs the client
    * during the setup() stage.
    * 
    */
    getStartService: () => Promise<InternalDynamicConfigServiceStart>;
}

export type DynamicConfigServiceSetup = Pick<InternalDynamicConfigServiceSetup, 'registerDynamicConfigClientFactory' | 'registerAsyncLocalStoreRequestHeader'>;

Between setup() and start()

To make configs configurable to the admin, CRUDL routes have to be registered. However, if the HTTP route registration is handled in setup() stage, there will be a circular dependency since httpServer now depends on the dynamicConfigService and the dynamicConfigService depends on httpService. As a workaround, the dynamicConfigService will have an intermediate step registerRoutesAndHandlers() (tentatively named) to register http routes and other handlers after http finishes setting up. This serves two purposes: to register http routes and the async local store in the preAuth server extension. This is what enables a per request context to be feasible:

public async registerRoutesAndHandlers(setupDeps: RegisterHTTPSetupDeps) {  
    const { http } = setupDeps;

    registerRoutes({
        http: http,
        ...
    });

    this.#logger.info('registering middleware');  
    // Register the async local storage with all the registered localStorage headers
    http.server.ext('onPreAuth', (request, h) => {  
        const localStore = createLocalStore(this.#logger, request, this.#requestHeaders);  
        this.#asyncLocalStorage.enterWith(localStore);  
        return h.continue;  
    });  
}

start()

In start(), the service will create the readonly client for other core services and plugins to consume. If a custom client factory was provided, the custom client will be created. Otherwise, the default OpenSearch client will be created. When start(), finishes the service will expose the following methods:

export interface InternalDynamicConfigServiceStart {  
    /**
    * For core services ONLY. This will expose a read-only client to read config values
    */
    getClient: () => IDynamicConfigurationClient;  
    /**
    * For core services ONLY. This will expose a read-only async local store map to pass into the client
    */
    getAsyncLocalStore: () => AsyncLocalStorageContext | undefined;  
}

// No DynamicConfigServiceStart will be exposed to plugins

DynamicConfigService should be the first service to start so that other core services can use its client.

Example usage

For plugins

// Getting the asyncLocalStore
const store = context.core.dynamicConfig.asyncLocalStore;

// Getting the dynamic config client
const client = context.core.dynamicConfig.client;

// For plugins with a config path in its manifest file
const configObject = await client.getConfig({ configPath: "INSERT-PLUGIN-CONFIG-PATH" }, { asyncLocalStorageContext: store!});

// For core services/plugins without a config path in its manifest file
const configObject = await client.getConfig({ name: "INSERT-PLUGIN-ID-OR-CORE-SERVICE" }, { asyncLocalStorageContext: store!});

// Accessing the value
const bar = configObject.foo;

For core services

// Getting the asyncLocalStore
const store = internalDynamicConfigServiceStart.getAsyncLocalStore();

// Getting the dynamic config client
const client = internalDynamicConfigServiceStart.getClient();

// For plugins with a config path in its manifest file
const configObject = await client.getConfig({ configPath: "INSERT-PLUGIN-CONFIG-PATH" }, { asyncLocalStorageContext: store!});

// For core services/plugins without a config path in its manifest file
const configObject = await client.getConfig({ name: "INSERT-PLUGIN-ID-OR-CORE-SERVICE" }, { asyncLocalStorageContext: store!});

// Accessing the value
const bar = configObject.foo;

Challenges

Scoping the client for plugins

Currently, all configs are isolated per plugin. This means that plugin A cannot access plugin B without B explicitly exposing its configs. How ConfigService handled config object scoping was leveraging the PluginInitializerContext to expose a create function that would create the scoped down client. Since plugins can register their own custom config store client in the setup() stage and the client can only be used in the request context, the initializer context cannot be used to create the scoped down client.

Core service limitations

While core services can have configs, they cannot be disabled meaning DynamicConfigService will exist as a permanent service. This can cause excess network calls to OpenSearch if a Dashboards instance does not use this feature. A workaround could be to define some "dummy" client that only returns the configs from the ConfigService and set that as the client if the service is "disabled".

Alternatives

A plugin was initially considered to make the service easy to enable/disable and prevent an extensive core change. However, plugins cannot access all configs because configs are isolated per plugin. Additionally, core services cannot be changed as core services cannot depend on a plugin (circular dependency).

[seraphjiang]

@xinruiba @BionIT @zhyuanqi @zengyan-amazon @SuZhou-Joe @ruanyl @kavilla would you help to review

[xinruiba]

Thanks @huyaboo for this RFC!

Based on this RFC, we are planning to create a dynamicConfigService in core-service to handle all core-services and plugin configurations. This approach makes sense. Looking ahead, to make core-service and plugin configurations more flexible, I see an opportunity for improvement.

What if we create a dynamicClientConfig service like you have proposed, and at the same time, we also create a ConfigurationPlugin?

The dynamicClientConfig would be responsible solely for core service configurations, while the ConfigurationPlugin would manage the configurations for all other plugins.

This approach offers the following benefits:

1. Decouples configurations for core services and plugins.
2. Provides the potential for more customization in plugin configurations.

What do you think?

[BionIT]

Thanks @huyaboo ! Like the idea, a few questions and things to consider and happy to hear your thoughts:

1. for the config store, what the index would look like and how we plan to address deprecated configs or any changes on the config path that may lead to incorrect mapping of the new config schema vs the ones stored in the config store
2. I think the static config and dynamic config would have some overlap and the dynamic config would take precedence, is it correct?
3. since the client for the dynamic config is read only, do we plan to expose setters somewhere to set the values into config store?
4. from a user perspective, we might need to expose a few interfaces/APIs for user to CRUDL the dynamic configs, otherwise

[SuZhou-Joe]

Integrate with dashboard admin

It seems plugins can only get the client through context, I do not know if that can work with dashboard admin implementation because we try to detect if user is a dashboard admin in postAuth lifeCycle. @yubonluo could you please help to confirm that?

Performance concern

As the dynamicConfigService will be a global change and plugin will call the client method every time there is a incoming request. Have we consider a cache solution to mitigate the issue?

[yubonluo]

Integrate with dashboard admin

It seems plugins can only get the client through context, I do not know if that can work with dashboard admin implementation because we try to detect if user is a dashboard admin in postAuth lifeCycle. @yubonluo could you please help to confirm that?

Performance concern

As the dynamicConfigService will be a global change and plugin will call the client method every time there is a incoming request. Have we consider a cache solution to mitigate the issue?

Thanks @huyaboo ~ As mentioned above, we need to determine whether the users/groups is dashboard admin through request (code). If possible, cloud you provide a service to enable plugins to register, and a method to get the client through request? Here is an example:

private setupPermission(core: CoreSetup) {
    core.http.registerOnPostAuth(async (request, response, toolkit) => {
      const { groups = [], users = [] } = this.permissionControl!.getPrincipalsFromRequest(request));
      const client = dynamicService.getClient(request);
      const [configGroups, configUsers] = await client.getConfig(this.globalConfig$);
      updateDashboardAdminStateForRequest(request, groups, users, configGroups, configUsers);
      return toolkit.next();
    });
  }

[huyaboo]

Thanks @huyaboo for this RFC!

Based on this RFC, we are planning to create a dynamicConfigService in core-service to handle all core-services and plugin configurations. This approach makes sense. Looking ahead, to make core-service and plugin configurations more flexible, I see an opportunity for improvement.

What if we create a dynamicClientConfig service like you have proposed, and at the same time, we also create a ConfigurationPlugin?

The dynamicClientConfig would be responsible solely for core service configurations, while the ConfigurationPlugin would manage the configurations for all other plugins.

This approach offers the following benefits:

1. Decouples configurations for core services and plugins.

2. Provides the potential for more customization in plugin configurations.

What do you think?

@xinruiba The issue with the ConfigurationPlugin is that it does not have access to all the configs without a core service providing the whole config. There could be some concerns with dynamicConfigService exposing the entire config.

2. Provides the potential for more customization in plugin configurations.

I am curious what do you mean by this?

[xinruiba]

Thanks @huyaboo for this RFC! Based on this RFC, we are planning to create a dynamicConfigService in core-service to handle all core-services and plugin configurations. This approach makes sense. Looking ahead, to make core-service and plugin configurations more flexible, I see an opportunity for improvement. What if we create a dynamicClientConfig service like you have proposed, and at the same time, we also create a ConfigurationPlugin? The dynamicClientConfig would be responsible solely for core service configurations, while the ConfigurationPlugin would manage the configurations for all other plugins. This approach offers the following benefits:

1. Decouples configurations for core services and plugins.

2. Provides the potential for more customization in plugin configurations.

What do you think?

@xinruiba The issue with the ConfigurationPlugin is that it does not have access to all the configs without a core service providing the whole config. There could be some concerns with dynamicConfigService exposing the entire config.

2. Provides the potential for more customization in plugin configurations.

I am curious what do you mean by this?

Thanks for the response~ It refers to

1. Using dynamicConfigService to expose plugin configs to ConfigurationPlugin
2. Using ConfigurationPlugin to dynamic handle plugin's configurations

[huyaboo]

Thanks @huyaboo ! Like the idea, a few questions and things to consider and happy to hear your thoughts:

1. for the config store, what the index would look like and how we plan to address deprecated configs or any changes on the config path that may lead to incorrect mapping of the new config schema vs the ones stored in the config store

2. I think the static config and dynamic config would have some overlap and the dynamic config would take precedence, is it correct?

3. since the client for the dynamic config is read only, do we plan to expose setters somewhere to set the values into config store?

4. from a user perspective, we might need to expose a few interfaces/APIs for user to CRUDL the dynamic configs, otherwise

Thanks @BionIT

1. We can mimic the old config service and how it applied deprecations. I may need some investigation into how deprecations are handled but there deprecations to rename paths, make paths unused, and custom defined deprecations.
2. Yes. The logic will "merge" the two config blobs such that every corresponding value in the dynamicConfigBlob will override the old config blob
3. Yes, there should be a read-only client exposed to core services/plugins, which is a wrapper around the InternalClient, a client that adds create/delete operations. These operations will only be invoked via the CRUD routes.
4. Yes, this is why the registerRoutesAndHandlers() method is within the dynamicConfigService. This function will register CRUD routes (/config/create, /config/delete, etc.) and in the future, we can have a management page that calls these routes to change configs directly from UI

[zhyuanqi]

Hi Huy. Thanks for the purpose. So the purpose of dynamic configuration service is to enable config change without a restart. I have some questions here.

1. When customer provides a incorrect config, how do we handle it. Will it crash the request call?
2. How will access to the dynamic configuration store be secured?
3. When the external config store is unavailable or a network issue, what will happen?
4. How frequently will configurations be fetched from the external store, and is there a caching mechanism in place?
5. I am a bit confused on registerRoutesAndHandlers() method. So for each request, we use dynamicConfigService to fetch values from the external config service?

[huyaboo]

@yubonluo Not sure if we can include the client using the request, but we may be able to expose getStartServices() in the DynamicConfigServiceSetup, a method that returns a promise of the start services (currently, it's only for the InternalDynamicConfigServiceSetup). Since your use case depends on a postAuth handler and the asyncLocalStore will be ready on preAuth, it may be possible to do something like the following:

private setupPermission(core: CoreSetup) {
    core.http.registerOnPostAuth(async (request, response, toolkit) => {
      const { groups = [], users = [] } = this.permissionControl!.getPrincipalsFromRequest(request));

      // Get the config store configs
      const startServices: DynamicConfigServiceStart = await core.dynamicConfigService.getStartServices();
      const client = startServices.getClient();
      const store = startServices.getAsyncLocalStore();
      const updatedWorkspaceConfigObject = await client.getConfig({ name: 'workspace' }, { asyncLocalStoreContext: store! });

     /**
     Assume the following config schema

     schema.object({
          enabled: schema.boolean({ defaultValue: false }),
          configGroups: schema.arrayOf( ... ),
          configUsers: schema.arrayOf( ... ),
     });

     */
     const { configGroups, configUsers } = updatedWorkspaceConfigObject;     

      updateDashboardAdminStateForRequest(request, groups, users, configGroups, configUsers);
      return toolkit.next();
    });
  }

As the dynamicConfigService will be a global change and plugin will call the client method every time there is a incoming request. Have we consider a cache solution to mitigate the issue?

@SuZhou-Joe Cache will have to be baked into the config store client implementation. For the default OpenSearch config store client, the configs will be cached by their configPath with a refeshInterval to clear the cache. The reason the cache implementation has to be implemented within the client is that different clients can be cached by different values.

[huyaboo]

Hi Huy. Thanks for the purpose. So the purpose of dynamic configuration service is to enable config change without a restart. I have some questions here.

1. When customer provides a incorrect config, how do we handle it. Will it crash the request call?

2. How will access to the dynamic configuration store be secured?

3. When the external config store is unavailable or a network issue, what will happen?

4. How frequently will configurations be fetched from the external store, and is there a caching mechanism in place?

5. I am a bit confused on registerRoutesAndHandlers() method. So for each request, we use dynamicConfigService to fetch values from the external config service?

@zhyuanqi

1. Ideally, the create operation will perform a validation check so it will reject invalid entries. We can leverage the schemas registered when plugins are discovered by calling schema.validate() on the updated config entry
2. I'm not sure about security/roles and who has access
3. We can set a timeout for the external config store in the client. There is a fallback with the client that will just take values from the yml file if necessary.
4. If plugins/core services want to onboard their configs to be dynamic, then on every request, the client will be called. Caching I've mentioned in this comment https://github.com/opensearch-project/OpenSearch-Dashboards/issues/7111#issuecomment-2195756693
5. The purpose of registerRoutesAndHandlers() is to register CRUD routes (/config/create, /config/delete) and create the async local store. This store will help group together relevant headers into one place and prevent props drilling.