Introduce hook for lock/install_deps provider

[lpil]

I wish to run some code after deps have been fetched but before the Erlang code in the deps have been compiled into beam files.

Specifically I wish to run the Gleam compiler in order to generate Erlang source files from Gleam source files. If the compiler is run as a hook on compile any new Erlang source files created for the deps is not detected by rebar3 and thus never gets compiled into beam files.

I cannot use the rebar3 compiler behaviour as it assumes that the input can be compiled on a file-by-file basis, while Gleam needs to be run on the project as a whole for its static type inference.

We had a brief discussion on IRC and another hook was suggested a solution here. If this is the right way to go I would like to implement this with some guidance so I can do it in a suitable fashion up to the standards of the rebar3 team :)

Thank you!

[ferd]

I think it would make sense yeah.

One thing I'm realizing right now is that we need to specify the behaviour of the hook being called recursively. In this case I think we should not allow it, for the simple reason that it doesn't make sense for a dependency that has a lock hook to have it executed every time by its parent applications.

We'd have to make sure the hook is only executed for the entire run, as with releases, so that we don't end up calling the hook recursively within deps.

That, on the other hand, would mean that the hook will have to be specified manually in each project and each dependency that wants to build with gleam; you'll never be able to get an automated build by just including deps aside from shipping pre-built binaries.

I don't see another way for this to work though. This is otherwise too fundamental a difference in build tool models to reconcile, I think.

[lpil]

For my use case I only wish for the hook to be called once, for the top level project.

It is a bit unfortunate that it needs to be added to the top level for each Gleam project. I was hoping to find a way to write a rebar3 plugin that would be able add the hook if required but I've been so far unable to work out if this was impossible, though I've not yet spent much time learning how rebar3 works.

For Erlang and Elixir projects including hex packages written in Gleam I am thinking that the best way to support them is to have the hex package contain the precompiled Erlang files generated by the Gleam compiler, that way they don't need the compiler at all. I think this is possibly a different matter to this issue

[ferd]

The only way I can think you could add the hook dynamically is if you were to call your own gleam compiler plugin and have it inject the state into the arguments loaded from disk, but generally the workflow is the opposite: rebar3 gives each provider the config it requires with the profiles applied.

The way Rebar3 is structured, each provider knows its own dependencies. compile calls lock which calls install_deps, which calls app_discover. If you write your own plugin (say gleam_compile), you can use your plugin to inject data and then have it call the compile provider explicitly, and you can even replace the top-level compile task for Erlang with yours by using a project plugin. The thing you can't do though is have your plugin inject itself as a dependency of compile so that it can modify the state it will run on.

When you write a hook, it runs before or after the task you hook into, but does not have the ability to inject configuration changes. We simply decided long ago that it would become very risky and brittle over time to have people just magically change config options unannounced.

[lpil]

Ah! So it sounds like in the scenario of a a person writing Gleam code it would be a better idea to write a gleam_compile plugin and add that as a dependency to their rebar3 project. It would replace the default compile provider with one that runs the Gleam compiler before calling compile, which would compile the deps and top level Erlang code as required.

Does that sound good to you?

It doesn't solve the problem of how Erlang projects would compile Gleam projects added as deps but I would like to defer that problem for later.

[ferd]

Yeah. That one would work pretty straightforward I think. The only difference between a regular plugin and project_plugin is that a plugin declared in project_plugin is allowed to override a default rebar3 command, but only when at the root of the project (i.e. a dep can't replace a parent's compile command)

[lpil]

That's great! Thank you very much!

Do you know of any project plugins I can take a look at?

[ferd]

I believe https://github.com/vernemq/rebar3_cuttlefish is one of them. It's a regular plugin, it's just that instead of registering a new name (or a similar name under a new namespace), it explicitly registers the name of an already existing command. Declaring it as a project_plugin as a user makes rebar3 know it's safe in that current context to override the existing providers.

[lpil]

Thank you

[lpil]

I've been testing out this idea in this repo. My provider can be found here: https://github.com/gleam-lang/rebar_gleam/pull/4/files#diff-bd8261164402ee47af0d144178042231

When I run rebar3 compile my provider in the test project my provider is called, but when another task that depends on compile such as rebar3 eunit it seems that my version does not get called.

Is that the intended behaviour or have I made a mistake? I'm looking to always have my version be called so the Gleam code is always compiled.

[ferd]

The EUnit provider is special; it does not actually rely on compile to work but on lock directly: https://github.com/erlang/rebar3/blob/master/src/rebar_prv_eunit.erl#L19 -- it manually calls the Erlang compiler provider: https://github.com/erlang/rebar3/blob/master/src/rebar_prv_eunit.erl#L340-L341

This is a hack that was needed because it manually injects eunit_opts into erl_opts in order to keep things working the way they were in rebar 2.x among other things. You will find that the ct compiler ends up doing something similar.

[lpil]

Brilliant, thank you once again :)