Improved offline builds for rebar3

[tsloughter]

Background

TODO

Tasks

TODO

[KennethL]

We want to bring up the importance of this and try to find a solution for this as soon as possible. We can implement the necessary changes and additions as soon as we agree on the way forward.

[KennethL]

I suggest to put this on the agenda for the Dec 7:th meeting

[ferd]

Ran a quick Demo with John.

The short version of it: Rebar3 can seemingly support the type of vendoring the Ericsson team requires in the following way:

1. Declare an umbrella project (with code in apps/<libname>)
2. Declare all external dependencies in each of the top-level apps' rebar.config file (e.g. apps/myapp/rebar.config contains {deps, [elli, recon]}) but no dependencies are declared in the root's rebar.config file (these definitions get inherited by all sub-applications, and when we do the vendor step, it creates a cycle)
3. call rebar3 get-deps to fetch but not build dependencies. These should now be in _build/default/lib
4. mkdir lib at the root, and then mv _build/default/lib/<app-to-vendor> lib/
5. to upgrade libraries, remove it from lib/ and use rebar3 upgrade <appname> like usual to fetch a new build, and move it back into the lib/ directory

This will make the vendored app be tracked locally in source and rebar3 will not even attempt to fetch it.

The caveats we have:

• This does not prevent unvendored app from being fetched (whether with hex or git), which should be done with capabilities or possibly by further patches
• We'll need to generate a rebar3 build step that fetches deps, modifies its own structure to support that vendor format, and adds it to build artifacts so one can just download a vendored rebar3 build (useful for package maintainers)
• It's possible this scheme is gonna fail on plugins, and it will require us to add support to load local plugins (as project_plugins to be safe) from a local directory the way we support doing it from libraries.

[josevalim]

Is it worth making the _build/default/lib/ bit configurable so it always downloads to lib anyway?

[ferd]

Not really, because the trick is that the build is still in _build, that's where the artifacts go. The copy/move bit just lets you transform dependencies into source applications easily by letting rebar3 fetch them for you from wherever. In fact, for git deps this might also create .git directory if you used a git resource for it.

[starbelly]

This does not prevent unvendored app from being fetched (whether with hex or git), which should be done with capabilities or possibly by further patches

FWIW, the way mix and mix hex works could work for rebar3. Specifically, all the hex related odds and ends could be moved into rebar3_hex. A convenience task could be added to install rebar3_hex or explain how to add it. Likewise, and in the vein of mix, if you try to get external deps, you are prompted if you want to install hex. This would of course be a major breaking change I believe.

[ferd]

With vendoring and the offline switches added to the bootstrap process, I believe this is now decently covered.