ENV variable replacement with nested configs in releases

GalaxyGorilla commented 6 months ago

Environment

Rebar3 report version 3.22.1 generated at 2024-04-30T14:30:15+00:00

Operating System: x86_64-pc-linux-gnu ERTS: Erlang/OTP 26 [erts-14.2.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns] Root Directory: /home/sascha/.asdf/installs/erlang/26.2.3 Library directory: /home/sascha/.asdf/installs/erlang/26.2.3/lib

Loaded Applications: bbmustache: 1.12.2 certifi: 2.11.0 cf: 0.3.1 common_test: 1.26.1 compiler: 8.4.2 crypto: 5.4.1 cth_readable: 1.5.1 dialyzer: 5.1.2 edoc: 1.2.1 erlware_commons: 1.6.0 eunit: 2.9 eunit_formatters: 0.5.0 getopt: 1.0.2 inets: 9.1 kernel: 9.2.2 providers: 1.9.0 public_key: 1.15.1 relx: 4.8.0 sasl: 4.2.1 snmp: 5.15 ssl_verify_fun: 1.1.6 stdlib: 5.2.1 syntax_tools: 3.1 tools: 3.6

Escript path: /home/sascha/.asdf/installs/rebar/3.22.1/bin/rebar3 Providers: app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report repos shell state tar tree unlock update upgrade upgrade upgrade vendor version xref

Current behaviour

I'm want to use nested release configurations (see https://www.erlang.org/doc/man/config.html) such that a release specific sys.config.src imports a common.config.src. The latter contains e.g. logger configuration which is shared between different releases. The sys.config.src looks like this:

[
    "common.config.src",
    {some_other_app, [
        [...]
    ]}
].

You can see with .src config that I'm using OS environment variables for runtime configuration in BOTH files. This works without problems when only the sys.config.src contains environment OS variables.

The error I get varies depending on if I use the name common.config.src:

Could not start kernel pid (application_controller) ("error in config file \"./common.config.src.config\" (none): configuration file not found")

% note the additional weird `.config` suffix in the error

or if I use common.config, which does simply not resolve any variables (which leads to config errors).

Expected behaviour

Nested configurations with OS environment variable replacement should work :).

I'm not certain if that's even a rebar3 issue, but any comment on that would be nice.

ferd commented 6 months ago

That will be tricky, because supporting that requires a bit of custom code that is really portable, and so far we do it with just a little awk hook in https://github.com/tsloughter/relx/blob/57c0d31b91737e55ab73fcdb0f2c7f1224ff944a/priv/templates/extended_bin#L239-L246 for unix-like systems (I think we never supported it for Windows).

so what we'd end up needing here is a shell script that recursively expands files and variables to support it (because it runs on an unspecified system, and the complexity has never warranted this effort so far)

GalaxyGorilla commented 5 months ago

Just out of curiosity, if I or someone else would work on this: how do you ensure portability of those scripts? ShellCheck'ing the hell out of it, using some reference shells?

ferd commented 5 months ago

Yeah probably. We've so far mostly crossed our fingers by using the least controversial/advanced set of features possible and avoiding changing things too hard

erlang / rebar3