Plan refactor of secret and non-secret configuration data

[thekaveman]

The existing Django models capture both secret configuration items and non-secret configuration items. The vast majority of fields are non-secret and do not require additional controls.

For the secret configuration, values are currently maintained in Azure Key Vault and read into a new app service environment each time the app restarts (e.g. with a new deploy, or if an engineer restarts the app manually during debugging).

We want to continue to maintain the majority non-secret configuration data simply and directly via Django models and stored in the Django database.

We need a plan for how to incorporate the secret values in an environment where we do not re-create the database each time the app restarts, and thus would not (necessarily) read from Key Vault at app startup time.

Acceptance Criteria

• [ ] Secret configuration continues to be maintained in Azure Key Vault
• [ ] Non-secret configuration is moved out of Azure Key Vault and is managed directly in the Django admin interface
• [ ] Secret configuration is not visible within the Django admin interface (yet -- need roles/permissions etc.)
• [ ] Engineers with appropriate access can update secret values in Azure Key Vault and apply the changes to the existing app deployment, perhaps requiring an app restart, but without requiring a new deployment

Additional context

• 1313

• Benefits app config field mapping (March 2023)

Activities

Running list of activities related to this effort

• [x] Revisit and update the Benefits app config field mapping (March 2023) -- :sparkles: UPDATED tab :sparkles:
• [x] Determine how to programatically read secret values
• [x] Determine how to refactor model classes with secrets

Implementation plan

• [ ] Implement a simple helper script in the spirit of this one in data-infra; use Azure's python libraries to read a value from a Key Vault by secret name. Ensure the Key Vault / environment (e.g. dev, test, prod) is not hardcoded; create a setting from an environment variable or similar.
• [ ] Create a custom Django field type that inherits from models.TextField. The default field implementation stores the name of the secret. An instance method like get_secret_value() uses the helper script to get the value of the named secret on-demand.
• [ ] Using the field mapping spreadsheet, update all secret fields to have this new field type. Update the field names to include _secret_name
• [ ] Add new properties to each model that use the helper script above and the _secret_name field value to get the actual value of the secret. These properties should be named the same as the original (pre-refactor) secret field (e.g. without the _secret_name postfix)

[thekaveman]

Maybe helpful here: https://learn.microsoft.com/en-us/python/api/overview/azure/keyvault-secrets-readme?view=azure-python&source=recommendations

[thekaveman]

Could follow this pattern / helper function get_secret_by_name: https://github.com/cal-itp/data-infra/blob/main/packages/calitp-data-infra/calitp_data_infra/auth.py

[thekaveman]

We talked through this plan at a recent Dev Workshop, and made some adjustments based on the conversation. Those are reflected in the steps above.