rebar3 clean destroys prepackaged beams

[tokenrove]

So, it is possible to have dependencies that are binary-only; that is, they contain only ebin and not src. rebar3 clean in projects with such dependencies breaks the build, because rebar doesn't know how to get the deleted beams back.

rebar 3.1.0+build.3415.ref5ccaaa8 on Erlang/OTP 18 Erts 7.3.1

[ferd]

Well that is one big problem since I'm not sure how we can have a provider whose sole functionality is 'delete the beams' figure out when not to delete the beams.

[vans163]

@tokenrove If the dependency is binary only I assume there is only a versioning need for source control. In that case what if you just place the dep (the .beams) into a root level folder like _staticdeps and pass -pz _staticdeps to the erlang vm when it loads.

@ferd Does this sound like an adequate solution? If so rebar3 could support this like it does _checkouts.

No matter if the .beam is in a ebin folder or directly in _staticdeps or any folder heirarcy. Anything inside _staticdeps that is a .beam will never be cleaned?

[ferd]

There's no reason to have special magic deps like that IMO. There has to be a nicer solution than that, because we'd need to know they're binary to automatically place them there and then we could reuse the information in clean instead.

[vans163]

Well I see the magic because say company A buys from company B erlang "work", company B says they cant give the source so they package up the .beam without any debug info, perhaps even encrypt it, and sell it to company A. So company A all they have is a binary and maybe a version number. If company A is lucky they will also have some documentation.

The only way to include this .beam sanely is to just have it there, and not touch it. So essentially it is indeed a magic dep, and it just has to be there, in any way it can. Outside of source control, outside anything.

The problem gets tougher though, as its outside an application structure, also if there are any accompanying nifs. Anyways food for thought.

It could always be made the case that, if you need to use these kinds of static beams, you should make the .app yourself and place it correctly, that would also make nifs work. Then cleans can ignore .beams and .so/.dll in that app. Or maybe add a rebar.config you can put like 'ignore_clean'. So just inside that app, cleans will not remove .beam and .so/.dll. Afaik priv dirs are already ignored from clean, and .so/.dll go into priv.

[ferd]

What happens in that case is that usually the beam will be installed at the system level and added to your ERL_LIBS path to be available globally.

If you really need it as a dep, I think we'd need to think of a new dep type rather than a new dep location for things, and finding ways to preserve the .beam files from there. Hooks could also be used. But just adding a new metadirectory is not the good way for wards, I think.

[tokenrove]

I feel like rebar should actually track what it builds (see, for example, ninja and how it handles -t clean) instead of blindly destroying beams, but I doubt that's going to happen.

In the specific case that prompted this bug, I moved away from the binary dependency (which was, indeed, version controlled) where possible, and wrote more aggressive make clean rules where it wasn't possible, so feel free to close this, although I suspect more people will run into the same issue eventually.

[tsloughter]

I believe @talentdeficit has a 'next gen' compile provider that does track what it creates.

[talentdeficit]

the current compiler is mostly carried over from rebar2 and has a number of limitations including a lack of tracking of non .erl files and a lack of extensibility

i have a prototype of a meta compiler that takes a list of compilers (that implement the meta compiler's behaviour) and generates a single digraph of all dependencies (including intermediate generated files, ie a .thrift file generates a .hrl file that an .erl file depends on). this means that we both get accurate tracking of what files were produced by rebar3 and which were part of the project and that we can accurately reduce the graph to just files that need recompiling and files that depend upon them

a beam compiler which just copies a precompiled file from the project tree into the build tree might look like:

-module(rebar3_mc_precompiled).

-export([inputs/1, dependencies/1, context/1, compile/1]).

inputs(App) ->
  %% return all files in the app dir that are beams. this should probably only consider
  %% `src_dirs` and `extra_src_dirs`
  AppDir = rebar_app_info:dir(App),
  rebar_utils:find_files(AppDir, "*.beam").

dependencies(_App) ->
  %% no dependencies
  fun(_Input) -> [] end.

context(_App) ->
  %% normally we'd return an opaque term here that encapsulates things like compiler
  %% options, env vars, command line options, etc that is stored in the digraph and can
  %% be used to detect when the file needs to be recompiled via a simple comparsion,
  %% but there are no inputs for a simple file copy so this is essentially ignored
  ok.

compile(App) ->
  %% copy the beam file from it's place in the project dir to the equivalent
  %% path in the output dir, returning the path to the resulting file
  AppDir = rebar_app_info:dir(App),
  OutDir = rebar_app_info:out_dir(App),
  fun(Input) ->
    Dest = rebar_file_utils:rebase_file(Input, AppDir, OutDir),
    ok = rebar_file_utils:cp_r(Input, Dest),
    Dest
  end.

the meta compiler works by first calling the inputs callback of all enabled compilers to generate a list of files to compile, the context callback to get their compile time context, the dependencies callback to get the function that determines dependencies of the file and the compile callback to get the function which compiles the inputs

then it builds a graph of input files and their dependencies, storing the context alongside the inputs in the graph. the graph is flattened into a list where all dependencies are compiled before their dependents. any files who have unsatisfied dependencies are also stripped from the list. the compiler is called for each file in the remaining list, the inputs that were compiled marked as so in the graph and the outputs collected. the meta compiler then runs again over the list of outputs adding them to the graph and recomputing the list of files to compile. this is repeated until either no new inputs are returned or the graph contains only files with unsatisfied dependencies (either a missing dependency or a circular dependency)

i still have some work to do on it but i'll hopefully have more time to work on it now the summer is over. initially i plan to only support a subset of the files types currently handled by the rebar compiler (precompiled beams, hrl and erl) and add support for elixir and lfe files so it won't be backwards compatible (altho it should be able to handle the majority of existing rebar projects) so it will need to be packaged as a plugin but hopefully then the community can add compilers that work on other files types including yrl, xrl, asn1, thrift, protobuf and other file types and we can eventually reach a point where we can consider replacing the existing compiler