plugins and various erlang versions

[asmodehn]

Hi,

I have been using rebar, erlang and elixir via asdf for a little while now...

I seem to have issues when switching erlang versions:

=ERROR REPORT==== 13-Dec-2022::13:05:07.778850 ===
beam/beam_load.c(162): Error loading module rebar3_run:
  This BEAM file was compiled for a later version of the run-time system than 24.
  To fix this, please recompile this module with an 24 compiler.
  (Use of opcode 178; this emulator supports only up to 176.)

=ERROR REPORT==== 13-Dec-2022::13:05:07.778856 ===
Loading of /home/alexv/.cache/rebar3/plugins/rebar3_run/ebin/rebar3_run.beam failed: badfile

From what I understand, plugins where compiled with otp v25, and I am now using v24.

I have a few questions:

• is there something obvious that I am missing ?
• is there a quick immediate fix for that (I dont see a clean/compile commands for rebar3 plugins...)
• is there a proper fix, that would prevent this problem from occuring in the first place (not using the cache, relying on asdf uuid of environment, following elixir by adding the otp version in the string, or some other idea ?)

Overall I d like to be able to switch rebar and erlang versions independently, or make the link with the OTP version somewhat more explicit... Thank you for any information.

[Stratus3D]

Unfortunately I don't have a good answer to any of these.

• is there something obvious that I am missing ?

No, if you compile a rebar version against OTP 25 and then switch to OTP 24 it will break as you have observed.

• is there a quick immediate fix for that (I don't see a clean/compile commands for rebar3 plugins...)

No. The only "fix" is to uninstall and re-build with the rebar version with the OTP version you are currently using, defeating part of the point of the version manager.

• is there a proper fix, that would prevent this problem from occuring in the first place (not using the cache, relying on asdf uuid of environment, following elixir by adding the otp version in the string, or some other idea ?)

I've thought about this some, and there are a couple ways I could solve this:

• As you pointed out, follow what is done for precompiled Elixir versions by adding the OTP version at the Rebar version number. This would work, but I'd be introducing my own versioning scheme on top of what Rebar already has. Not necessarily a problem, but also not the sort of complexity I want users to have to think about.
• The other solution to this would be to add support to asdf core for explicit dependencies between plugins. So this case we'd someone indicate (via a plugin callback that the user wouldn't have to know about) that this rebar plugin depends on the erlang plugin. So any change to an Erlang version would require a re-build of the rebar version. Internally we'd store each rebar build a directory name containing a hash of all dependency versions, followed by the actual rebar version. Something like 3e4c4ad-3.16.1. Then when the current Erlang version changes asdf would instead look for rebar in say ac18c3d-3.16.1 and not find it, and would trigger a message to the user saying they need to run asdf install again.

I'm inclined to go with that last option, as this is something that comes up a lot with asdf. There is of course a lot of work to make that happen, and some performance implications as well. Thoughts?

[asmodehn]

Well, for context I also read https://github.com/asdf-vm/asdf/issues/1127 and https://github.com/asdf-vm/asdf/issues/196.

I am also in favor of avoiding the complexity of a full dependency graph resolution in asdf (ordering in .tool-versions is "good enough" imho). Afterall, I use asdf as it is easier to setup and maintain than a full nixpkg setup, yet allows version management of various tools with a simple interface.

I like the idea of version tags, or having another version, but I don't really see an easy way to support it in asdf-vm in a generic way without seeing it blow up and become much complex than anticipated (the usual fate of package management systems). And leaving it to each plugin means the user needs to think and manage these...

So:

• I noticed that I didn't have any problem with rebar3 itself, but with plugins...
• I also noticed that part of rebar3 documentation:

Do note that if you are planning to work with multiple Erlang versions on the same machine, you will want to build Rebar3 with the oldest one of them. The 3 newest major Erlang releases are supported at any given time: if the newest version is OTP-24, building with versions as old as OTP-22 will be supported, and produce an executable that will work with those that follow.

I am not used to rebar3 much ( Ive just recently been trying to setup a C/Erlang/Elixir decent dev env), so I may be a bit off-track, but maybe the solution is in two parts:

• on asdf-rebar side: enforce this rule of 3 newest major release supported -> build it with the oldest supported one already installed. Maybe coding that check into the shim and forcing a rebuild if necessary would be enough ? Unless you are working on a rebar plugin, the erlang version you are using and the erlang version rebar was built with won't conflict, I m guessing ?

• on rebar3 itself : have a different way to manage plugin builds ? Like putting them in a location depending on some uuid depending on the build of rebar itself ? I see a bunch of issues on rebar regarding plugins... Or maybe... in asdf-rebar, via setting REBAR_CACHE_DIR with a hash depending on the rebar/erlang build ? That would sidestep the issue I think, as far as asdf is concerned, since a different rebar build would point to different plugins... But I also see : https://github.com/erlang/rebar3/issues/2762 so we might want to be careful playing with this.

So compared with your solution, I am thinking to make rebar build more generic, and keep the version hash trick for plugins only... Maybe less generic, but simple enough (no big asdf feature needed) ? Can you see any usecase in which that would be a no go ?

[Stratus3D]

Hi @asmodehn sorry for the late reply here.

• on asdf-rebar side: enforce this rule of 3 newest major release supported -> build it with the oldest supported one already installed. Maybe coding that check into the shim and forcing a rebuild if necessary would be enough ? Unless you are working on a rebar plugin, the erlang version you are using and the erlang version rebar was built with won't conflict, I m guessing ?

Currently the Erlang version used to build a particular version of Rebar isn't recorded anywhere. This information is only available at install time, as an install may be run from any directory, and the environment may be changed at any time. Doing so would be possible, and then we'd have an exec-env callback script that could maybe print an error when the version doesn't match (or isn't newer). I don't recall whether or not the exec-env callback has the ability to halt execution, which is what we'd ideally want in this cause.

on rebar3 itself : have a different way to manage plugin builds ? Like putting them in a location depending on some uuid depending on the build of rebar itself ? I see a bunch of issues on rebar regarding plugins... Or maybe... in asdf-rebar, via setting REBAR_CACHE_DIR with a hash depending on the rebar/erlang build ? That would sidestep the issue I think, as far as asdf is concerned, since a different rebar build would point to different plugins... But I also see : https://github.com/erlang/rebar3/issues/2762 so we might want to be careful playing with this.

REBAR_CACHE_DIR with a hash depending on the rebar/erlang build ? Hmm... I like the idea of that. However that issue you linked to does seem to indicate setting REBAR_CACHE_DIR would cause other issues for our users.

I guess I'm inclined to either do nothing, or go with the first option for now. Eventually we'll have some basic dependency management baked into asdf.

If we implemented the first option, this is how it might work:

• The bin/install script should capture the Erlang version number immediately before compiling Rebar3. Something like this - erl -eval 'erlang:display(erlang:system_info(otp_release)), halt().' -noshell. Write this version to a file somewhere in the install directory
• Create a bin/exec-env callback. The callback should do the following:
  • Get the current Erlang version (erl -eval 'erlang:display(erlang:system_info(otp_release)), halt().' -noshell).
  • Read the file containing the Erlang version used during the build.
  • If they match, let the script exit normally with no output. If they differ, print an error message indicating the Erlang build version doesn't match the current build version, and exit with a non-zero status code.

What do you think about this @asmodehn ? Feel free to take a stab at implementing this if you want. If once asdf itself supports dependency management this code could just be deleted.

[kivra-pauoli]

@Stratus3D, instead of building rebar3 locally (in this plugin), could we pick it up from the pre-compiled releases available in GitHub directly?

e.g. https://github.com/erlang/rebar3/releases/download/3.24.0/rebar3

AfaIk these are arch. agnostic and are compiled with at least OTP-2, which would partially prevent compatibility issues when using it locally with different Erlang versions.

e.g.

erlang 27.0.1
rebar3 3.24.0

will mean that that rebar3 is (potentially?) not compatible with say Erlang 26.2.5.3, but if we download the pre-compiled artifact not only is it faster at installing but that issue would go away.

[kivra-pauoli]

Another solution would be to create a rebar3 (specific) plugin, but this would:

1. not benefit the many installs already out in the wild
2. potentially cause confusion

(?)

In any case I think I'm gonna hack something locally and if results are promising I can pull request.