Create a consistent configuration cascade/design for existing configuration parameters

[xiamaz]

Is your feature request related to a problem? Please describe. Currently certain settings are set site wide, project wide or per user. This is often an issue, as in different scenarious configuration settings might need to be preset at different levels. A more consistent design is needed here to serve different needs that might arise.

Describe the solution you'd like Configurations should be available on site, project, user level. Each should override the previous one.

Describe alternatives you've considered Only defining configurations at site level might not be sufficient for customization.

Only defining configurations at user level does not allow projects to enforce that certain configurations are applied.

Additional context

[xiamaz]

@xiamaz Complete ticket and create more specific tickets for actual design.

[xiamaz]

General configuration design

Currently configuration parameters are defined in many different locations. In the future, a consistent interface should exist to define configuration parameters at the following levels:

• site
• project
• user

Implementation remarks

Configuration models should be defined once and be linked to either site level, project level or user level. When accessing configuration parameters, always the user level should be obtained and this will take project level configuration if no specific user configurations are defined.

[stolpeo]

Current Status

• SODAR core already provides settings on project and user level for each Django app.
• Site wide configuration is done via environment variables that only can be changed with a restart of the server. These settings are very basic.
• There is also a SODAR-core-independent project setting for query presets that are editable by all users which is located under the case list view.
• Gene panels can be defined site-wide which is a complex setting and thus located in an own app.
• SODAR core project and user settings are limited to Integer, String, Boolean which probably needs to be extended.
• The interface for SODAR core settings is rather limited

Open Ends

• Who should be able to edit the settings?
  • site => admin
  • project => pi/delegate
  • user => user
• How can we achieve more complex site-wide settings?
• SODAR core needs to be extended to at least FLOAT, and more complex data types or we detach the settings from SODAR core for this.
• How do we proceed with the existing settings (genepanel, query presets) that are scattered randomly throughout the application?

[xiamaz]

Who should be able to edit issues is clear now.

Way forward for complex data types is JSON, we just need to find the correct data type to store in SODAR core.

Configuration cascade is:

• site wide
• project-specific
• project-user specific
• user specific (all projects)

We should collect configuration parameters in documentation and define their configurability regarding these four aspects. New configuration parameters need to always update documentation first.

Currently open tasks:

• [x] create ticket in sodar-core regarding missing data types for configuration, but non-blocking
• [ ] maximum string length for string configuration parameters needs to be known
• [ ] create documentation on current configuration parameters and their configurability

[stolpeo]

RE: datatypes in SODAR core: https://github.com/bihealth/sodar-core/issues/1439

[stolpeo]

Currently existing configurations and where they live

Site-wide (environment variables - require server restart)

• VARFISH_* (they are not intended to be set as superuser and won't interfere with any other settings a user might make)

Site-wide (SiteAppPlugin)

• ~Genepanels - user menu - superuser (create/edit) - user (view) - own models for storing values~ not a setting per definition
• VueApp Settings - user menu - superuser - link refers to entrypoint
• ~Beacon site - user menu - superuser - own models for storing values~ not a setting per definition

Project-specific (ProjectAppPlugin scope PROJECT)

• User defined tags to attach to cases - project menu (app: variants) - owner/delegate
• Disable pedigree sex check - project menu (app: variants) - owner/delegate
• Exclude from in-house database - project menu (app: variants) - owner/delegate
• Ts/Tv valid range - project menu (app: variants) - owner/delegate

Project-specific (other)

• Quick Presets - case list - all users

Project-user-specific (ProjectAppPlugin scope PROJECT_USER)

• None

User-specific (SiteAppPlugin or ProjectAppPlugin scope USER)

• UMD Predictor API Token - user settings menu (app: variants)
• GA4GH Beacon Network Widget - user settings menu (app: variants)
• Changelog seen in version - user settings menu (app: variants)
• Variant Filtration Inline Help - user settings menu (siteapp vueapp settings)
• Variant Filtration UI Complexity - user settings menu (siteapp vueapp settings)

[stolpeo]

• Currently the quick presets (project-specific) violates the constraint that it should be editable only by pi/delegate + it is in the wrong place.
• The site-wide vueapp settings menu does not exist, assuming it is just a dummy to hold the user specific app settings

[stolpeo]

RE: datatypes in SODAR core: apparently JSON already exists. String max length is 255 but will be bumped in the next version.

[stolpeo]

Editing SODAR core app settings is possible via the AppSettingAPI. This allows us to build an interface around that. The more complex SiteApp settings can receive a link-out.

from projectroles.app_settings import AppSettingAPI
app_settings = AppSettingAPI()
app_settings.get('app_name', 'setting_name', project_object)

[stolpeo]

Technically all settings can that can fit in a JSON dict can be registered as settings in an ProjectAppPlugin or SiteAppPlugin. The hierarchy can be derived from the scope. Currently there are no interfering settings where this would come into play.

[stolpeo]

Implementation draft

• Introduce a new model under the cases app. I would refrain from introducing a new app because the siteapp and projectapp plugins are already bloated.

  class Configuration(models.Model):
  scope = selection[SITE/PROJECT/PROJECT_USER/USER]
  name = CharField

• Any model can be registered as configuration such that holds a ForeignKey to the new Configuration model.
• SiteAppPlugin and ProjectAppPlugins are scanned for app_settings but not registered in the model.
• A new view lists all configurations registered with Configuration (currently this only would be QuickPresets, GenePanels and BeaconSite) and all app_settings that are registered with a SiteAppPlugin or ProjectAppPlugin.
• To change the settings that are coming from SODAR core, we use the AppSettingsAPI.
• To ensure the cascade, implement a function that returns the correct setting that must be used to access settings.
• [ ] double check if it is possible to have the same app_setting name for a setting with different scope.

[holtgrewe]

Dimensions of sodar-core configuration:

1. Where does a setting name/key live? Choices: site-wide app, or project-app
2. What is the scope? Choices: project, user, project AND user
   • site-wide app (always implicit: project=None)
     • user=obj => editable by user
     • user=None => global configuration setting editable by admin only
   • project-wide app
     • project=None, user=obj => editable by user
     • project=obj, user=obj => editable by user
     • project=obj, user=None => editable by project owner/delegate

What are default values not stored in the database?

1. Factory defaults hard-coded in Python (change requires container rebuild & redeploy)
2. Overrides for factory defaults configured in environment variables (change requires server restart)

[xiamaz]

Discussion 27.06.2024

• Suffixed scope names
• potentially present to mikko and maybe adopt to sodar-core
• estimated remaining work ca 1 workday

[stolpeo]

I just pushed the first implementation of the cascaded configuration parameter design (WIP).

It is based on parameter names in SODAR core app_settings that end on

__site | __user | __project_user | __user

This allows us to use the same parameter name throughout all levels and establish a cascade of what value is returned, depending if a parameter is being set on a certain level. The cascade is as follows:

site < user < project < project_user

• A site setting is set by the admins.
• A user can override this setting, if available in the users scope.
• A project setting is set by the project owner or delegate, is specific to a project and overrides site and user settings, in case the setting is shared in both scopes.
• A user can override this setting for a specific project if available in the users scope.

Currently, it is accessible through an REST API which defines 6 endpoints:

• settings/api/site/
• settings/api/user/
• settings/api/project/{project_uuid}/
• settings/api/project-user/{project_uuid}/
• settings/api/all/
• settings/api/all/{project_uuid}/

All endpoints except the api/all ones have also PUT and PATCH methods and can take a ?a=reset with the GET which deletes all settings in the scope which will make it return default values set in the app_settings. The api/all endpoints only return values and differentiate whether a project uuid has been passed or not. In that case it will return project-specific and project-user-specific values, otherwise defaults.

The Settings API has different functions to return setting objects to:

• get, set, reset and get default parameters on every scope level
• get all settings

For now the implementation is stalled because I need more input:

What remains is when a parameter is configured across all scopes, when should it return a setting and when a default? The SODAR core AppSettingsAPI returns a value when a parameter is queried, either the set value or the default. It is only distinguishable with a trick (looking up the existence in the AppSetting table).

I could construct something that looks whether a parameter is set on a level and if not return the default of the lowest (least worth) level possible, but I don't know if this intended or if we currently run into something that is not solvable easily. Maybe parameters shouldn't live across all scope levels, or only project and project-user are connected and site and user.

@holtgrewe @xiamaz

[xiamaz]

Discussion 18.07.2024

• MH: we need use-cases that need to be collected.
• MZ: use cases
  • filter presets (high priority)
  • user-specified gene panels (medium priority)
  • quality check metrics (low priority)
  • single setting parameters (low priority)

First focus on improving filter presets:

• presets are currently bound to SODAR projects, presets from all visible projects are visible
• presets can be used from other existing projects
• presets will have versions (draft, active, archived)

After discussion, this can be closed.

Post-mortem: complex tickets only with very specific use cases in the future