Support backticked environment variables in levels of config

[robbieaverill]

In SilverStripe 4 we can use backticked environment variables in YAML configuration files, but only if they're under an Injector config definition, e.g.:

SilverStripe\Core\Injector\Injector:
  MyServiceClass:
    properties:
      MyProperty: '`MY_ENV_VAR`'

This will have Injector call ->setMyProperty() with the parsed value of MY_ENV_VAR when loaded.

This is a documented thing, but still something that developers get tripped up on, because you assume it would also apply to other parts of YAML configuration. Example: https://github.com/silverstripe/silverstripe-framework/issues/7987 and https://stackoverflow.com/questions/53437811/silverstripe-env-variable-value-in-config/53459699#53459699

We could add the ability for the configuration layer to parse all environment variables when they're referred to during collection, so that something like this would work too:

MyServiceClass:
  my_config_prop: '`MY_ENV_VAR`'

So that when MyServiceClass calls $this->config()->get('my_config_prop') it returns the parsed value of MY_ENV_VAR from the environment.

Acceptance critreria

• [ ] Backtick variables are always converted to the matching Environment variable
• [ ] For injector YML definition
• [ ] For Private static properties outside the injector context

Note

• Either we mention in the DOC that backtick can not be used as regular characters in YML or we find a way to escape them.

[dnsl48]

I'd suggest to pass that information through YAML tags rather than through backticks. symfony/yaml that we use supports it, so it may look like that:

MyServiceClass:
  my_config_prop: !env 'MY_ENV_VAR'

That would be backward compatible with the existing configs and we wouldn't have to apply some magic rules for all the string values in the configs. That also would be extensible and potentially in the future we might add support for more tags (e.g. something like !service for '%$...').

[tractorcow]

Since environment variables can be modified at runtime, but config is cached at boot, you'll need to be extremely careful that you don't rely on env vars during boot that aren't set yet (or that are subsequently changed).

I suggest if you implement such a config resolution strategy, that you make it run-time not cache-time. (i.e. make it a middleware that evaluates on Config::inst()->get() similar to how extensions work). The literal back-ticked value would be cached.

[OldStarchy]

@dnsl48 I am against using !env 'MY_ENV_VAR' as it shows an error in my editor "unknown tag <!env>". Even if the tag is available / defined elsewhere, a false positive in my error list would bother me to no end.

[patricknelson]

TL;DR: A much more concise version of this is here: https://github.com/silverstripe/silverstripe-framework/issues/7987#issuecomment-908811168. For anyone interested in reading a novel, carry on below. 😅

---

I came here wanting something similar myself. But, in my case, I not only wanted to see my environment variables interpolated in my YAML config, I also wanted to consolidate the definition of lots of environment-specific configuration into YAML instead of having them all in PHP constants, since I'm still on SS v3 and I need to eventually migrate to SS v4. Also in the interim, I need to work in containers & Kubernetes environments that rely more on environment variables instead of constants, which are extremely difficult to manage when you have a large number of environments (we have like 5 environments where we've deployed code, which I've coined "deployment environments"). So basically I needed to solve these problems:

• How to consolidate configuration across 5 environments so that only secrets are uncommitted?
• How do I move to environment variables when using SS v3 which relies so heavily on constants?
• How do I consistently reference development values which can be committed consistently (e.g. database server name) with those values which cannot be committed and are secret (e.g. database user/pass combo) while also using environment variables to share these values in the container (e.g. when running bash scripts and etc)?

Unfortunately, SS v3 relies very heavily on _ss_environment.php but I built a forward-compatible polyfill that will not only allow you to use .env easily in SS v3 but also will distribute commitable environment variables into Environment as well as interpolate them into YAML config for consistent access regardless of if you're using .env to initialize environment variables or if they're actually first-class environment variables (like those defined in docker-compose.yml or Kubernetes manifests).

If anyone's interested in that code, please let me know and I could work on a gist. But in the meantime, here's a VERY basic example of that YAML structure and the supporting PHP code responsible for initializing those environment variables which are germane to this particular issue, in case it helps!

Example YAML and supporting PHP code (excludes Globals class that actually do the heavy lifting of initializing, the _ss_environment.php polyfill that loads .env variables and the copied SS v4 Environment classes which holds these environment variables). :neckbeard:

---
##
## Contains default site configuration for ALL environments and overridden on a per deployment-environment basis. All
## other environment configs will run after this one thanks to the '#mysite-defaults' entry in their respective headers.
##
## This leverages SilverStripe's built-in YAML configuration capability. For more details on how this works, please see:
## https://docs.silverstripe.org/en/4/developer_guides/configuration/configuration/
##
Name: mysite-defaults
After:
  - 'framework/*'
  - 'cms/*'
---
Globals:
  ##
  ## NON-SECRET environment variables which are automatically loaded and accessed as environment variables throughout
  ## the codebase via the Environment::getEnv() method. This is for any variables that aren't secret and therefore can
  ## still be committed to code. This is useful since we have so many environments.
  ##
  ## These are automatically initialized by Globals::init().
  ##
  ## IMPORTANT:
  ##
  ## 1. Please only override the necessary values in the environment-specific YAMLs.
  ##
  ## 2. Actual environment variables and values defined in ".env" will take precedence over any values defined in YAML.
  ##
  ## 3. Variables that MUST be defined as actual environment variables are only *documented* here and must be placed
  ##    either in .env or defined as an environment variable (depending on hosting environment) so that they are never
  ##    committed. For example, all documented but non-committed environment variables should just be COMMENTS following
  ##    this format:
  ##
  ##      Secrets:
  ##      #ENV_VAR_NAME="SECRET"
  ##
  ##      Secrets with non-secret defaults (i.e. only secret if overridden):
  ##      ENV_VAR_NAME: 'value' # SECRET IF OVERRIDDEN
  ##
  ##      Non-Secrets:
  ##      #ENV_VAR_NAME=VARIES
  ##
  ##    This allows them to be easily copied into .env and then modified.
  ##
  ## 4. All defaults defined here are DEVELOPMENT/TESTING DEFAULTS. Production values must be defined elsewhere!
  ##
  Environment:
    # The special environment variable corresponding with this specific deployment (e.g. 'prod', 'uat', 'qa', 'dev',
    # 'vm' and etc). All possible values can be found in the BASE_DOMAINS constant. All domains/subdomains are then
    # derived from this initial value, as well as all subsequent automated configurations.
    #
    # IMPORTANT: Every environment MUST define this environment variable. Also, the "deployment environment" setting
    # here is not to be confused with the "SilverStripe environment" setting.
    # TODO: Should most these notes move to the .env.example file?
    #DEPLOYMENT_ENVIRONMENT=VARIES

    # Location where SilverStripe caches (class manifests, templates, and other cached values) should be stored.
    # If empty (default), the cache will be stored under the website root as 'silverstripe-cache'. Please be sure to
    # override this in development to prevent caches from being stored in website directory shared with the host machine.
    # IMPORTANT: If used, this value MUST be defined as an actual environment variable due to how early it is needed!
    #TEMP_PATH=VARIES

    # SilverStripe environment type: dev, test, live
    SS_ENVIRONMENT_TYPE: 'dev'

    #####################
    ## DATABASE CONFIG ##
    #####################

    # Non-secret database values
    SS_DATABASE_SERVER: ''
    SS_DATABASE_NAME: ''
    #SS_DATABASE_USERNAME="SECRET"
    #SS_DATABASE_PASSWORD="SECRET"

    # Delegate readonly queries to another server, enable this and set the PORT. All other settings will be copied from
    # the main database configuration, however: You can also define an alternative server, DB name, username and password.
    # See all constants and where this is controlled: CustomMySQLDatabase
    SS_READONLY_DATABASE_ENABLED: false
    SS_READONLY_DATABASE_PORT: ''

    #########################
    ## DEFAULT ADMIN LOGIN ##
    #########################

    # Default CMS admin username and password.
    #SS_DEFAULT_ADMIN_USERNAME="SECRET"
    #SS_DEFAULT_ADMIN_PASSWORD="SECRET"

  ##
  ## Generic and globally accessible values. These are *effectively* constants that can vary per-environment,
  ## committed code and can be overridden on a per deployment-environment basis. When accessed via Globals::constant()),
  ## these values can also contain secrets imported from environment variables as well (useful for consistency and
  ## reusability).
  ##
  ## NOTE: You can access these values via either:
  ##
  ## 1. Globals::constant('CONSTANT_NAME') - Strongly recommended for all site code, since it is not easily
  ##    inspected using IDE intellisense. It also ensures developers know to look in this file for the definitions.
  ##
  ## 2. CONSTANT_NAME - As an actual constant. However, this is discouraged as it's only intended for SilverStripe's internals.
  ##
  Constants:
    # See above. Must be defined as constants, so interpolate environment values now.
    SS_DEFAULT_ADMIN_USERNAME: '`SS_DEFAULT_ADMIN_USERNAME`'
    SS_DEFAULT_ADMIN_PASSWORD: '`SS_DEFAULT_ADMIN_PASSWORD`'

Then somewhere in the _config.php I define:

<?php
use SilverStripe\Core\Environment;

/**
 * Do things that would normally be done in SilverStripe v3's ConfigureFromEnv.php
 * 
 * TODO: SSv4: This entire closure can be removed as it's handled automatically in v4.
 */
call_user_func(function() {

    // Initialize SilverStripe's environment type.
    Config::inst()->update('Director', 'environment_type', Environment::getEnv('SS_ENVIRONMENT_TYPE'));

    // Initialize database confirmation.
    global $databaseConfig;
    $databaseConfig = array(
        "type" => "MySQLDatabase", // NOTE: Will use Injector to pull our custom class instead.
        "server" => Environment::getEnv('SS_DATABASE_SERVER'),
        "username" => Environment::getEnv('SS_DATABASE_USERNAME'),
        "password" => Environment::getEnv('SS_DATABASE_PASSWORD'),
        "database" => Environment::getEnv('SS_DATABASE_NAME'),
    );
    DatabaseAdapterRegistry::autoconfigure();

    // Allow a default user/pass in non-live environments. Also normally performed in v3's ConfigureFromEnv.php.
    if(!Director::isLive() && defined('SS_DEFAULT_ADMIN_USERNAME')) {
        if(!defined('SS_DEFAULT_ADMIN_PASSWORD')) {
            user_error("SS_DEFAULT_ADMIN_PASSWORD must be defined in your _ss_environment.php,"
                . "if SS_DEFAULT_ADMIN_USERNAME is defined.  See "
                . "http://doc.silverstripe.org/framework/en/topics/environment-management for more information",
                E_USER_ERROR);
        }
        Security::setDefaultAdmin(SS_DEFAULT_ADMIN_USERNAME, SS_DEFAULT_ADMIN_PASSWORD);
    }
});

[GuySartorelli]

The relevant documentation doesn't specify use in Injector config as a requirement, and I have heard anecdotes that it was previously available for any config value. Arguably the fact that it doesn't work is a bug.

[robbieaverill]

Perhaps things have changed since the issue was opened. The reason for the behaviour was that Injector was responsible for injecting environment variables at runtime into strings that reference them rather than the confit manifest. Some work was required for the confit manifest to support this functionality instead.

[robbieaverill]

Those linked docs say this, which maybe indicates the change was deliberate in 4.2:

Environment variables cannot be used outside of Injector config as of version 4.2.

[patricknelson]

FWIW, my opinion on this has also evolved over the past year. My original goal was to somehow accomplish interpolation of environment variables anywhere in configuration (e.g. via backtick references) and not just under Injector config definitions such as in the original description above.

IMHO, really all I ultimately ended up needing and using was a combination of:

1. The envorconstant exclusionary rule under Only which is a standard supported built-in that can look at one or more environment variables (more on this below)
2. A DEPLOYMENT_ENVIRONMENT environment variable, which varies per “environment” upon which I deploy my application (I could have 5 or more, far more than SS_ENVIRONMENT_TYPE alone allows) and just define that in .env
3. Use an uncommitted .env for any secrets and for that one DEPLOYMENT_ENVIRONMENT variable.
4. Dynamically augment environment variables from this per-environment YAML config via some initialization in PHP (see also, below).

Then, I just created one config yml for each deployment environment. Any non-secret environment variables can be defined under a some main key (like Environment in my comment above) and are just initialized early into Environment::setEnv() via manual iteration through the array of config values.

// Initialize environment variables, making sure not to override any that may already be configured.
// IMPORTANT: Falsey values (empty strings, literal false, etc) may not appear at all in this array.
$envVars = static::config()->Environment ?: [];
foreach($envVars as $envName => $envVal) {
    if (Environment::getEnv($envName) === false) {
        Environment::setEnv($envName, $envVal);
    }
}

Then, the developer has the option to manually override (especially temporarily for local development or in emergency scenarios) values committed to .yml by editing the current environment’s .env file. So far this has worked out well for me with no need at all for any sort of interpolation.

EDIT: p.s. Any situations where manual interpolation of environment variables would be needed in Config have been edge case (or really, non existent) enough for me to be fine with sticking with the approach of using something like Class::config()->Variable = ... or Config::inst()->update('Class', 'Variable ', ...) to solve that problem instead.

[patricknelson]

TL;DR: In favor of not proceeding with this at all and instead utilizing the envorconstant exclusionary rule under Only for per-environment configuration, or, using Class::config()->Variable = ... or Config::inst()->update('Class', 'Variable ', ...) in situations where interpolation (beyond what is already possible) is truly required. 😊

[maxime-rainville]

This has to be merged before the CMS5 beta to get in CMS5.

[GuySartorelli]

This didn't make it in CMS 5 - we can look again for 6.