Implement Standardized Environment Variable Validation

[k-allagbe]

Description

The goal is to standardize the way we validate environment variables and configurations.

We need to implement a solution that performs this validation for all critical environment variables, checking aspects such as type, format (e.g., regex for URLs, email addresses), presence of required variables, etc.

Acceptance Criteria

1. All critical environment variables should be validated before the server or application starts.
2. The validation should support various data types (string, integer, boolean, etc.).
3. Regular expressions should be used to validate variables like URLs and email addresses.
4. The solution should halt the application start-up if invalid environment variables are detected.

Notes

• This validation should happen as early as possible during the application initialization to prevent any invalid configuration from affecting the system.

[rngadam]

How I envision it:

a declarative solution with an env.yml that:

• contains the list of environment variables to be validated

  • and for each environment variable
  • their type (email, number, etc)
  • if they are optional or not
  • if they can be empty
  • optional validation using regexp

  I would be astonished if this functionality didn't already exist as an opensource python library somewhere so some research is in order.

[k-allagbe]

The closest tool i could find is datamodel_code_generator. But it is a command line tool. I could try and create our own tool for this, based on Pydantic.

[rngadam]

@k-allagbe please outline a bit more of how it would look like with Pydantic

[k-allagbe]

@rngadam https://github.com/ai-cfia/membrane-backend/commit/adc44102ceb7c3017202c56b4185755f003611b4 A drawback is that the yaml file would follow Pydantic's constraints naming. It would not be language agnostic.

[rngadam]

@k-allagbe best that you can a Draft PR so I can start commenting on it early.

are we also able to integrate text description for each field and links to references for the doc? and generate configuration documentation from the env.yaml?

[k-allagbe]

@rngadam https://github.com/ai-cfia/membrane-backend/pull/78 There is a description field available for that indeed. It would look like this:

MEMBRANE_FRONTEND:
  type: str
  field_info:
    default: http://localhost:3000
    description: <desc>

[rngadam]

ok, but my concern is we decided to both document 1) a description of the field and 2) a link to the dependency documentation. are we able to integrate both here with a single description field?