REBAR_CACHE_DIR and global config clash

[eproxus]

When setting REBAR_CACHE_DIR the internal config of global_rebar_dir get's set with that prefix (see apps/rebar/src/rebar3.erl:427).

Some things that are looking for global configuration will actually end up in REBAR_CACHE_DIR/.config/rebar3/... which is wrong (in my opinion).

One example where this becomes a problem would be the templating system, which uses rebar_dir:template_dir(State). That function in turn eventually gets the value for global_rebar_dir, as set above. E.g. if REBAR_CACHE_DIR is set to ~/.cache/rebar3 templates are looked for in ~/.cache/rebar3/.config/rebar3/templates which is incorrect.

There is a workaround, which is setting REBAR_GLOBAL_CONFIG_DIR too. That variable has it's own issues, in that it needs to be set to $HOME (and not $HOME/.config/rebar3 which one would think, so that's probably another bug).

There is also an internal config called global_cache_dir but if rebar3.erl is changed to set that instead, it also doesn't work correctly.

It seems settingglobar_rebar_dir if REBAR_CACHE_DIR is set is too heavy-handed, but a better implementation is not obvious to me. What would be the correct fix for these issues?

[jj1bdx]

Some things that are looking for global configuration will actually end up in REBAR_CACHE_DIR/.config/rebar3/... which is wrong (in my opinion).

I faced this issue today when I tried to use rebar3 with kerl, where kerl defines the REBAR_CACHE_DIR environment variable specific for each OTP installation. The rebar3 new command didn't look up ~/.config/rebar3/templates, and it took a few hours for me to find out why by thoroughly reviewing the source code.

[paulo-ferraz-oliveira]

I looked at this, and reading @eproxus' comment, I'm as confused regarding what should be set. I do agree that it seems odd we're setting global_rebar_dir for REBAR_CACHE_DIR (this appears to be done twice), but I didn't dig much further.

[ferd]

Yeah I think I recall digging in slack, and the unfortunate state is that this stuff is all very inconsistent and will be hard to fix without breaking anyone's setup. This would need a full re-mapping and clean-up to re-define where the values are set, what they mean, and how they're used.

It's very possible right now that it was intended one way in a code area, and was used another one in another area, and that there's no way to fix this without willingly breaking either pattern in a specific context.

[jj1bdx]

One thing I'd like to suggest is that this behavior of ${REBAR_CACHE_DIR/.config/rebar3/templates and the other affected settings should be documented. I couldn't find the document.

[choptastic]

I just came across this issue as well (with an automated template install that wasn't being recognized), and my solution was to make a little plugin for outputting directories.

https://github.com/choptastic/rebar3_dirs and https://hex.pm/packages/rebar3_dirs

At least this way, I can use something that, should this functionality ever change, will hopefully remain backwards compatible, rather than having to know and then code around the internal logic for where the config directory ends up.

$ rebar3 dirs template_dir
/home/gumm/kerl/26.2.4/.cache/rebar3/.config/rebar3/templates

$ rebar3 dirs
base_dir: /home/gumm/code/myapp/_build/default
deps_dir: /home/gumm/code/myapp/_build/default/lib
root_dir: /home/gumm/code/myapp/.
checkouts_dir: /home/gumm/code/myapp/_checkouts
checkouts_out_dir: /home/gumm/code/myapp/_build/default/checkouts
plugins_dir: /home/gumm/code/myapp/_build/default/plugins
lib_dirs: apps/*lib/*.
project_plugin_dirs: plugins/*
global_config_dir: /home/gumm/kerl/26.2.4/.cache/rebar3/.config/rebar3
global_config: /home/gumm/kerl/26.2.4/.cache/rebar3/.config/rebar3/rebar.config
template_dir: /home/gumm/kerl/26.2.4/.cache/rebar3/.config/rebar3/templates

[eproxus]

@choptastic That should be built into Rebar itself! ❤️

[choptastic]

@choptastic That should be built into Rebar itself! ❤️

I certainly appreciate the appreciation.

I'll gladly submit a PR to rebar3 if the team wants.

[ferd]

I wouldn't be opposed to having this sort of stuff built in, though there's some potential confusion around what we do for subtasks and hooks' env vars: https://github.com/erlang/rebar3/blob/main/apps/rebar/src/rebar_env.erl

[JesseStimpson]

I'm facing a related issue, and I'm not sure what is the right way to proceed. We use kerl to install Erlang on multi-user systems, and the recent change involving REBAR_CACHE_DIR requires that those users have write access to the location in which kerl has installed the Erlang package.

===> Could not write to "/opt/kerl/lib/26.2.5/.cache/rebar3/hex". Please ensure the path is writeable.

The pain point for me as system administrator, is that I would rather not expose this path as writeable to the users on the system.

I believe before the change in kerl (#457), rebar would have used the default ~/.cache/rebar3/hex.

I'm unsure if my question belongs in the rebar project or kerl project, but discussion on this topic seems to be centered here for now.

As a temporary workaround, I've instructed users to use the following when activating a kerl installation:

. /opt/kerl/lib/26.2.5/activate
unset REBAR_CACHE_DIR
unset REBAR_PLT_DIR

[paulo-ferraz-oliveira]

@JesseStimpson, this might also be of interest: https://github.com/kerl/kerl/issues/495.

[ferd]

At this point I don't think it is possible to solve this, we just made too many inconsistent choices within Rebar3 over the years. The only thing that seems reasonable is to map out all the use cases and breakage points, and pick and choose what is a clearer future behavior, and whatever option we pick will be backwards incompatible for some users and use cases.

[tsloughter]

Ugh, was global_rebar_dir inherited from rebar2? It seems off...

[tsloughter]

Also was meaning forever to move templates to XDG_DATA_HOME or ~/.local/share/rebar3