rebar3 plugins are already unloaded when execution reaches compile

[hurkatolto]

Pre-Check

Environment

Rebar3 report version 3.4.4+build.3913.ref387b89a4 generated at 2017-11-14T17:30:49+00:00

Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)

Task: Entered as:

---

Operating System: x86_64-apple-darwin16.6.0 ERTS: Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false] [dtrace] Root Directory: /usr/local/Cellar/erlang@19/19.3/lib/erlang Library directory: /usr/local/Cellar/erlang@19/19.3/lib/erlang/lib

Loaded Applications: bbmustache: 1.3.0 certifi: 2.0.0 cf: 0.2.2 common_test: 1.14 compiler: 7.0.4 crypto: 3.7.3 cth_readable: 1.3.0 dialyzer: 3.1 edoc: 0.8.1 erlware_commons: 1.0.0 eunit: 2.3.2 eunit_formatters: 0.4.0 getopt: 0.8.2 inets: 6.3.6 kernel: 5.2 providers: 1.6.0 public_key: 1.4 relx: 3.23.1 sasl: 3.0.3 snmp: 5.2.5 ssl_verify_fun: 1.1.2 stdlib: 3.3 syntax_tools: 2.1.1 tools: 2.9.1

---

Escript path: /Users/tothlac/mbin/rebar3 Providers: app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report shell state tar tree unlock update upgrade upgrade upgrade version xref

Current behaviour

I was writing a plugin similar to this one: https://github.com/kellymclaughlin/rebar3-tidy-deps-plugin/blob/master/README.md

Everything is similar except I was using the three original resource ids. E.g. git instead of github.

rebar3 compile fails with undef error and with following stacktrace:

[{rebar_github_resource,make_vsn,[“/Users/tothlac/work/rebarex”],[]},
 {rebar_utils,vcs_vsn,3,
              [{file,“/Users/tothlac/work/rebar3/_build/default/lib/rebar/src/rebar_utils.erl”},
               {line,665}]},
 {rebar_otp_app,preprocess,3,
                [{file,“/Users/tothlac/work/rebar3/_build/default/lib/rebar/src/rebar_otp_app.erl”},
                 {line,113}]},
 {rebar_otp_app,compile,2,
                [{file,“/Users/tothlac/work/rebar3/_build/default/lib/rebar/src/rebar_otp_app.erl”},
                 {line,49}]},
 {rebar_prv_compile,compile,3,
                    [{file,“/Users/tothlac/work/rebar3/_build/default/lib/rebar/src/rebar_prv_compile.erl”},
                     {line,139}]},
 {rebar_prv_compile,‘-copy_and_build_project_apps/3-lc$^2/1-2-’,3,
                    [{file,“/Users/tothlac/work/rebar3/_build/default/lib/rebar/src/rebar_prv_compile.erl”},
                     {line,99}]},
 {rebar_prv_compile,do,1,
                    [{file,“/Users/tothlac/work/rebar3/_build/default/lib/rebar/src/rebar_prv_compile.erl”},
                     {line,54}]},
 {rebar_core,do,2,
             [{file,“/Users/tothlac/work/rebar3/_build/default/lib/rebar/src/rebar_core.erl”},
              {line,154}]}]

That's because this line unloads all modules from code path: https://github.com/erlang/rebar3/blob/387b89a4cede139601cb517783e7cc97a3b917fd/src/rebar_prv_compile.erl#L40

The plugin will work only if I add the plugin repository to deps in rebar.config, without that I have this issue. It looks like all plugins are not available in the compile phase which can be also an issue if someone would like to write a plugin which is automatically called during compile.

Expected behaviour

Only the dependencies of plugins should be unloaded (those can version clash with the real dependencies of the project), but probably the plugin itself can stay in memory, or there should be an other workaround to fix the issue not by adding the plugin to the deps section.

[tothlac]

Let me explain the plugin which does not work because of the above mentioned problem. The intention is to automatically mirror github / pkg / hg repositories to our proprietary servers. So for example we have this section in rebar.config:

{substitute_urls,[{"git://github.com/${user}/${project}.git", "https://github.com/${user}/${project}.git"}]}.

{mirrorurls,[{"https://github.com/${user}/${project}.git", "https://our.bitbucket.server/scm/int/${user}${project}.git"}, ...]}.

We can't access github repos using git:// that's why we have the rule in substitute_url, so they will be converted to https:// addresses, but that's a simple substitution.

The other option mirrors the repository following the given rules. It will clone the repository every time the plugin encounters a repo which does not exist internally. If the repository exists internally, the plugin calls back into the original resource module (rebar_git_resource) but this timne the url in the source will point to the internal address.

Upgrading works as expected: if the internal repository does not contain some commits already present on github and there are commits in the internal repository these commits will be rebased against the ones available on github and if merging fails a PR is created containing the conflicting code.

We could solve it by directly adding some lines of modifications to the code of the existing resource modules calling out into our mirroring module, but it is a lot nicer to solve the task using a plugin.

I followed rebar_tidy_deps which adds a github resource to the resources, but in our case we want to directly steal all existing source types (git/pkg/hg) from the default modules.

After adding the plugin's resource modules the resources in rebar_state looks like this:

[{hg,rebar3_hg_mirroring}, {pkg,rebar3_pkg_mirroring}, {git,rebar3_git_mirroring}, {git,rebar_git_resource}, {pkg,rebar_pkg_resource}, {hg,rebar_hg_resource}]

This way rebar will automatically call the plugin's modules instead of the default ones since they go before the original ones.

The problem comes as it is described in the ticket when rebar3 tries to call ResourceModule:make_vsn/1 in the compile phase, and as it was previously unloaded by rebar_prv_compile it will fail with undef error.

One possible quickfix for the problem is when ResourceModule:make_vsn/1 is not available try to call the next resource module matching the resource type, like in this PR:

https://github.com/erlang/rebar3/pull/1663

Unfortunately putting the plugin into deps also does not work if the plugin is compiled after compiling some other dependencies.

Of course the long term solution is when reaching compile, rebar3 should unload only the dependencies of the plugins but not the plugins.

[ferd]

Yeah I think the good fix is to unload the plugins that are not also custom resources, or plugins that are not dependencies of an app. I do feel the former is safer because it prevents people from accidentally having their app depend on a plugin by accessing the plugin code without declaring a dependency (which would make future maintenance a nightmare)

The quickfix you suggest works really for just one resource provider here, so I'm hesitating to include it since it won't be very useful in the general case.

[ferd]

So uh, currently there's no way in the APIs in the way they were designed for us to obtain the path of a provider that is also a resource once that provider has been unloaded. I'm trying to figure out a decent way to be able to discover or track that path so that the compiler can load them as required before doing its thing, but it's a bit tricky doing that right now without doing deeper changes. Will keep this issue updated.

[tothlac]

Thanks for the answer. Would it be possible to have a temporary fix by merging the mentioned PR into master as we need to make this plugin work?