Docker - Build config.json from Environment Variables at runtime

Is your feature request related to a problem? Please describe. When using the Docker container, many config settings are unconfigurable during setup, and cannot be easily managed afterwards in environments that need high data portability (i.e. volumes).

Describe the solution you'd like Looking at the Gitea project, they use a standard format to allow all configuration options to be changed via environment variables. This means that any configuration changes can simply be done from a docker-compose file, rather than directly editing their configuration file. For example, you set environment variable GITEA__database__DB_TYPE=mysql, and this sets the corresponding config option:

[database]
DB_TYPE=mysql

Something like this could be done for MeshCentral. Obviously, MeshCentral uses json (whereas Gitea uses ini), so the exact tooling they are using can't be directly ported over, but conceptually the same ideas can be applied.

Additional context I'm currently working on a shell script-based parser using jq that would provide this capability. If there's interest, I'm happy to put together a pull request once I have it complete. The idea is this would replace the current startup.sh file.

In order to maintain compatibility with the existing container (and therefore, existing docker-compose files and settings), I'm imagining that it will work as follows:

If a config.json file does not exist, create one from the template.
Parse environment variables, and apply those to the config.json. Supported environment variables would retain the existing variables exactly as currently used and add variables of the format: MC__a__b__c... which would map to {"a":{"b":{"c": "value"}}}, for arbitrary depth.
After updating the config.json file with environmental variables, run MeshCentral as is currently done.

Of note, yaml supports multi-line strings, so it would be easy to make lists or other structures for the few config options that require it. For example, the webrtcConfig parameter contains iceServers, which is a list. You can set that in a docker-compose yaml file as:

environment:
  MC__settings__webrtcConfig__iceServers: > [
        { "urls": "stun:stun.services.mozilla.com" },
        { "urls": "stun:stun.l.google.com:19302" }
      ]

As I'm working through this, a couple more notes of issues and how I'm attempting to solve them. Any feedback on these approaches would be useful. I should have the pull request ready in the next week or so.

Empty Strings

One of the main keys in MeshCentral is the empty string domain key: "domains": { "": { ... } }. We need a way to represent this key that avoids ambiguity with other possible keys. In Gitea, they use the form _0Xaa_ to represent ASCII characters that aren't legal environment variable characters, where aa is the two letter ASCII code. For example, in Gitea, the key ui.admin has the period character, which is not a legal character in environment variables. As such, that key is represented as ui_0X46_admin.

For the empty string, we can use the null character 0X00. This would be a special interpretation, but is fairly sensical, and works well. That means our key "domains": {"": {"title": ...} } would be something like: MC__domains___0X00___title. Note the three underscores: that's two underscores, indicating a level, and the third underscore for the wrapper on the ASCII character code.

Case Sensitivity

I'm not sure if MeshCentral requires case sensitive keys in the config file, but fortunately, environment variables do not require upper case (it is merely by convention that environment variables are all upper-case). As such, values can be rendered exactly as they are in the name. For example, MC__domains would be "domains" and MC__DoMaInS would be "DoMaInS".

This also means that it's not trivial to check for case errors in the environment variables. My approach here is to simply assume the user is casing everything correctly, and blindly incorporate that into the config.json file.

_Keys (removed config keys)

By convention, Javascript uses _ in front of a name to indicate that it's private. In MeshCentral, we use this convention in the config file to "remove" a variable so that it is not applied in the config.

We can use a special value of _ to indicate that a variable should be disabled. For example, MC__domains=_ would cause the existing "domains": { ... } key to be replaced with a "_domains": { ... } key, preserving the existing value under this new _ key.

We can also recognize empty environment variables, and set the value to the empty string. For example, MC__domains= will preserve the "domains" key, but set the value to "", like: "domains": "".

Other Thoughts

This concept was mentioned in #22, so I would hope there's some interest in this setup. In that thread, two tools were suggested that can perform this action using Javascript/Typescript. That may be a better approach to this problem, but I'm woefully unskilled in either of those two languages, so I'm sticking with Shell for this.

After looking through the Dockerfile, I realized that python3 is currently installed as part of the container, which would make Python a first-class citizen in MeshCentral. However, I'm not convinced that Python is actually a requirement for MeshCentral, since I found mentions only in a few workflow pieces, and nothing in the actual code-base. It would seem that python3 is an unnecessary package for the container, and should be removed. As such, I'll hold off on a Python version of this until I hear otherwise that Python is going to stick around.

Ylianst / MeshCentral