Make API Keys & Access Keys out of serverConfig or ProxyAuthConfig configurable via Env Vars

[t83714]

Make API Keys & Access Keys out of serverConfig or ProxyAuthConfig configurable via Env Vars

It's a good practice that separate API Keys & Access Keys config from other non-credential configs and make them configurable via environment variables.

This also allows those credential configs to be stored in specialised storage designed for credentials.

e.g. on k8s cluster, it's a good practice to store credentials in k8s secrets.

this can be done with the yargs (or the dotenv-expand solution @zoran995 suggested) we used at this moment:

https://github.com/TerriaJS/terriajs-server/blob/d96739c4702383c2301f473768c52cb896fb8619/lib/options.js#L54

e.g.

.option("sessionSecret", {
            describe:
                "The secret to use to sign session cookies.  This can also be specified with the SESSION_SECRET environment variable.",
            type: "string",
            default:
                process.env.SESSION_SECRET ||
                process.env.npm_package_config_SESSION_SECRET,
            demand: true
        })

Acceptance Criteria

• Move out any API keys & Access Keys from ServerConfig and ProxyAuthConfig
• Make all API keys & Access Keys configurable via environment variables

[zoran995]

An idea, what about something like dotenv-expand just for JSON, still using .json config file but expanding values with environment variables on startup ($ENV_VAR is replaced). Have done this recently for another project and can share it if interested

[t83714]

@zoran995 Thanks for the suggestion and it might be a good idea 👍 As long as we can accept passing keys via env vars, we can keep the current config JSON structure internally. Probably can overwrite the config JSON fields with key values here: https://github.com/TerriaJS/terriajs-server/blob/d96739c4702383c2301f473768c52cb896fb8619/lib/options.js#L106 either manually or as you suggested simply using dotenv-expand.

[zoran995]

dotenv-expand was just a reference how they implemented it as their function already does the job of interpolating value, not sure if it is exported from dotenv-expand but we can take that function entirely in our codebase and maintain it for our needs (with referencing the original implementation)

Yeah that would be that 😀

I have implemented same thing here https://github.com/zoran995/geoportal-server/pull/14/commits/2dd909051ab8659dac63bad07380cccbcb4f51b6 in https://github.com/zoran995/geoportal-server/pull/14. Project is basically a rewrite of terriajs-server using NestJs architecture, I can try to get some time to create a PR here also

[joelsundholm]

For anyone reading this today and thinking about k8s deployment:

We use an initContainer with envsubst which writes the secrets into the config file which is in a mounted volume that we then mount to the main container. This way the actual TerriaJS image doesn't need to be cluttered or modified in any way.