Distribution packaging: allow to use system libraries

[Fnux]

Hello,

I am trying to package rebar3 for the Fedora linux distribution. When we build an erlang package, we obviously can't fetch archives from external services: I took a look at the provided documentation but did not find anything similar to rebar2's skip_deps=true and REBAR_DEPS_PREFER_LIBS="TRUE". Do we have similar features with rebar3 ? If not, how hard would it be to implement them ?

Thanks!

[ferd]

If you just use ERL_LIBS=path/to/lib/parent/dir and not declare them as dependencies in rebar3 it will run fine.

is the problem that in some cases you want to fetch them and in some cases you do not want to fetch them?

[Fnux]

is the problem that in some cases you want to fetch them and in some cases you do not want to fetch them?

Yep, the dependencies have to be shipped as distribution packages.

If you just use ERL_LIBS=path/to/lib/parent/dir and not declare them as dependencies in rebar3 it will run fine.

Since rebar3 will be used to build every erlang package shipped with the distribution, I don't think that removing the dependencies from rebar.config is a proper solution. Given some environment variable, rebar3 should be able to find the dependencies in ERL_LIB and detect the missing ones (and fail!).

[tsloughter]

This environment variable would also have to tell it to ignore the lock file, since it won't be able to verify the checksums from the lock file for the dependencies.

It is a bit odd to do. Maybe we'll have to... but yea.

[tsloughter]

Oh, but note that there is rebar3 bare compile which is used by mix to tell rebar3 to not resolve deps on its own. Maybe this is enough?

[Fnux]

It's not perfect but looks good enough for now. However, while rebar3 compile works fine rebar3 bare compile crash. I downloaded the precompiled binary from https://s3.amazonaws.com/rebar3/rebar3:

workdir/rebar3 » tar xf 3.6.1.tar.gz                       
workdir/rebar3 » cd rebar3-3.6.1 
rebar3/rebar3-3.6.1 » md5sum $(which rebar3)          
3010208afbe695b2fb5b3b6b2d14d74a  /home/fnux/.local/bin/rebar3
rebar3/rebar3-3.6.1 » reba3 bare compile
zsh: correct 'reba3' to 'rebar3' [nyae]? y
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> When submitting a bug report, please include the output of `rebar3 report "your command"`
rebar3/rebar3-3.6.1 » DEBUG=1 rebar3 bare compile
===> Expanded command sequence to be run: []
===> Provider: {default,do}
===> Expanded command sequence to be run: [{default,app_discovery},
                                                  {bare,compile}]
===> Provider: {default,app_discovery}
===> Provider: {bare,compile}
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: function_clause
===> Stack trace to the error location:
[{lists,reverse,[undefined],[{file,"lists.erl"},{line,147}]},
 {string,tokens,2,[{file,"string.erl"},{line,1801}]},
 {rebar_prv_bare_compile,do,1,
                         [{file,"/home/tristan/Devel/rebar3/_build/prod/lib/rebar/src/rebar_prv_bare_compile.erl"},
                          {line,45}]},
 {rebar_core,do,2,
             [{file,"/home/tristan/Devel/rebar3/_build/prod/lib/rebar/src/rebar_core.erl"},
              {line,154}]},
 {rebar_prv_do,do_tasks,2,
               [{file,"/home/tristan/Devel/rebar3/_build/prod/lib/rebar/src/rebar_prv_do.erl"},
                {line,68}]},
 {rebar_core,do,2,
             [{file,"/home/tristan/Devel/rebar3/_build/prod/lib/rebar/src/rebar_core.erl"},
              {line,154}]},
 {rebar3,main,1,
         [{file,"/home/tristan/Devel/rebar3/_build/prod/lib/rebar/src/rebar3.erl"},
          {line,66}]},
 {escript,run,2,[{file,"escript.erl"},{line,759}]}]
===> When submitting a bug report, please include the output of `rebar3 report "your command"`
rebar3/rebar3-3.6.1 » rebar3 report "rebar3 bare compile"
Rebar3 report
 version 3.6.1
 generated at 2018-08-19T12:34:32+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: rebar3
Entered as:
  rebar3 bare compile
-----------------
Operating System: x86_64-redhat-linux-gnu
ERTS: Erlang/OTP 20 [erts-9.3.3.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [kernel-poll:false]
Root Directory: /usr/lib64/erlang
Library directory: /usr/lib64/erlang/lib
-----------------
Loaded Applications:
bbmustache: 1.5.0
certifi: 2.0.0
cf: 0.2.2
common_test: 1.15.4
compiler: 7.1.5
crypto: 4.2.2
cth_readable: 1.4.2
dialyzer: 3.2.4
edoc: 0.9.2
erlware_commons: 1.2.0
eunit: 2.3.5
eunit_formatters: 0.5.0
getopt: 1.0.1
hipe: 3.17.1
inets: 6.5.2.2
kernel: 5.4.3.2
providers: 1.7.0
public_key: 1.5.2
relx: 3.26.0
sasl: 3.1.2
snmp: 5.2.11
ssl_verify_fun: 1.1.3
stdlib: 3.4.5
syntax_tools: 2.1.4.1
tools: 2.11.2

-----------------
Escript path: /home/fnux/.local/bin/rebar3
Providers:
  app_discovery as clean compile compile cover ct deps dialyzer do edoc escriptize eunit get-deps help install install_deps list lock new path pkgs release relup report shell state tar tree unlock update upgrade upgrade upgrade version xref 

I got the same result on a clean Fedora 28 system. Do you know what could be the source of the issue ?

[ferd]

yeah the bare compiler expects arguments, so since you passed none it crashed. It expects:

--paths [List Of Paths]
--separator   (separator between paths, defaults to a space character)

And it expects to run from the app's current directory

I'm sorry this isn't super pretty, there's not a whole lot of docs on it since mostly this has been used as an embedded command in mix for Elixir.

[Fnux]

Thank you! However, I bumped into a new issue: what am I supposed to feed to --paths ? I tried the lib directory of my erlang installation (and its childs: $mylib, $mylib/ebin, $mylub/include, ...) but wasn't able to get the -include_lib macro to find what I want. The ERL_LIBS environment variable did not help.

rebar3/rebar3-3.6.1 » ./rebar3 bare compile --paths /usr/lib64/erlang/lib           
===> Compiling rebar
===> Compiling src/rebar_prv_dialyzer.erl failed
src/rebar_prv_dialyzer.erl:13: can't find include lib "providers/include/providers.hrl"
src/rebar_prv_dialyzer.erl:95: undefined macro 'PRV_ERROR/1'

src/rebar_prv_dialyzer.erl:8: function do/1 undefined
src/rebar_prv_dialyzer.erl:84: spec for undefined function do/1
[...]

... the file exists!

rebar3/rebar3-3.6.1 » stat /usr/lib64/erlang/lib/providers-1.7.0/include/providers.hrl
  File: /usr/lib64/erlang/lib/providers-1.7.0/include/providers.hrl
  Size: 340         Blocks: 8          IO Block: 4096   regular file
Device: fd01h/64769d    Inode: 2767086     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:lib_t:s0
Access: 2018-08-20 17:35:29.744891884 +0200
Modify: 2017-11-17 03:25:47.000000000 +0100
Change: 2018-07-15 10:48:18.280963660 +0200
 Birth: -

http://erlang.org/doc/reference_manual/macros.html tells me that code:lib_dir/1 is used to find the library's directory:

rebar3/rebar3-3.6.1 » erl                                                             
Erlang/OTP 20 [erts-9.3.3.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.3.3.2  (abort with ^G)
1> code:lib_dir(providers).
"/usr/lib64/erlang/lib/providers-1.7.0"

~ » cat $(which erl)
#!/bin/sh
#
# %CopyrightBegin%
# 
# Copyright Ericsson AB 1996-2016. All Rights Reserved.
# 
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# 
# %CopyrightEnd%
#
ROOTDIR="/usr/lib64/erlang"
BINDIR=$ROOTDIR/erts-9.3.3.2/bin
EMU=beam
PROGNAME=${0##*/}
export EMU
export ROOTDIR
export BINDIR
export PROGNAME
exec "$BINDIR/erlexec" ${1+"$@"}

Did I miss something ?

[ferd]

Elixir folks call it as follows: https://github.com/elixir-lang/elixir/blob/1b152f2656ac2f529e3dd6695ee2de2d49460c96/lib/mix/lib/mix/tasks/deps.compile.ex#L185 which in their case seems to be a wildcard to all the ebin directories.

[Fnux]

I already saw this file, but it didn't work :/

[Fnux]

I was able to make it work, I don't remember what I did wrong when I tried in august.

[weiss]

FTR, I use a rebar.config.script such as the following:

% Set SKIP_DEPS=true (and ERL_LIBS) to skip dependency handling.

case os:getenv("SKIP_DEPS") of
    "true" ->
        SkipOpts = [deps, plugins],
        lists:foldl(fun(Opt, Acc) ->
                            lists:keydelete(Opt, 1, Acc)
                    end, CONFIG, SkipOpts);
    _ ->
        CONFIG
end.

[weiss]

I use a rebar.config.script such as the following: [...]

Ah, and the rebar.lock file obviously must be removed before building, as mentioned above.

FWIW, I now added some notes for distribution packagers to a project of mine in order to explain these things, but I can imagine it would improve their life if there was built-in support for skipping dependency handling.

[jubalh]

I'm working on Erlang packaging in openSUSE.

After reading this issue it seems to me that there is a way to make rebar3 use globally installed libraries/packages. But it sounds to me like some amount of effort is needed to achieve this.

I saw that the old rebar just had SKIP_DEPS=true as mentioned in this thread as well.

Is there any particular reason why such a solution wasn't implemented in rebar3?

[debalance]

What @jubalh is doing in openSUSE, I do in Debian, and I am wondering the same thing.

[tsloughter]

@jubalh @debalance you mean you work on adding distro packages of Erlang apps to the openSUSE and Debian repos or you develop Erlang apps and are trying to use the distro packages?

[debalance]

For me - the former. https://qa.debian.org/developer.php?login=debalance

[jubalh]

For me both :) But my question was about the former.

We already have some packages available. But most of them use legacy rebar and are not in latest version. I wanted to update them and generally improve our Erlang packaging state.

[joaohf]

Hello,

What @jubalh is doing in openSUSE, I do in Debian, and I am wondering the same thing.

And I am doing for YP/OE as well.

Some examples:

https://github.com/meta-erlang/meta-erlang/blob/master/recipes-devtools/rebar3/rebar3.inc https://github.com/meta-erlang/meta-erlang/blob/master/recipes-connectivity/ejabberd/ejabberd_20.04.bb https://github.com/meta-erlang/meta-erlang/blob/master/recipes-database/riak/riak.inc

But there, I decided to fetch erlang dependencies during the package bulding. That might not be a good practice for all scenarios.

[tsloughter]

Ok, cool.

I thought I'd start with the fact support for being able to use apps from ERL_LIBS as the source of deps is something I've been thinking we need to do to give an easier solution to people who don't want rebar3 to hit the internet. If we ever did that it would resolve your issue. But not everyone agrees on this and if it were to happen would probably be behind a flag.

But to be honest, we recommend everyone developing with Erlang (Elixir as well) to avoid distro Erlang packages. Not just packages of third party apps but also the packages of Erlang/OTP itself.

For the latter it has long been very confusing for users and a constant source of questions when they install their distro's package of Erlang and then discover parts don't work because what is considered part of Erlang/OTP is split out into separate packages by distros, like erlang-runtime_tools, erlang-ssl, erlang-public_key, etc.

So that is sort of the starting point for why distro package managers aren't a priority. Might be a different story if installing Erlang/OTP installed the full release of OTP :)

In rebar3, unlike rebar, we took a direction of providing more guarantees about the state at build time -- deps are automatically fetched when running rebar3 compile and locked in a lock file.

For development, as opposed to a user who is installing an application to use, I thought the standard used by all development tools of any language had pretty much become to have per-project dependencies.

Do you know if this is an issue other tools in other languages have had to address?

[jubalh]

For development, as opposed to a user who is installing an application to use, I thought the standard used by all development tools of any language had pretty much become to have per-project dependencies.

Unfortunately there is a constant discussion and reinvention of the wheel. Each new language starts out like this. Then packagers come a long and discussions happen for years. Until in some cases eventually there is a solution found.

In some cases (like for Rust packaging) distributions (at least openSUSE and some others that I checked) use vendoring using cargo vendor.

But it is quite ugly and more of a workaround really. Always the nice thing about distros was to have your libs installed and used dynamically. And once there is a security issue the maintainers need to just fix this one library. Now we need to gather all those packages that use vendored dependencies and update them. Various distributions create own solutions for this. And some do it manually. It all leads to a big mess actually.

Things like C, Python and so on work fine. Things like Rust are ugly. Go was ugly at first as well, but now is better. It really seems to be the same story all over again :-)

So that is sort of the starting point for why distro package managers aren't a priority. Might be a different story if installing Erlang/OTP installed the full release of OTP :)

AFAIK we install the full release of OTP. So at least we wouldn't add to this problem. And maybe a better way to fix this problem would be to convince distributions to do this be default.

I understand that regular developers might just want to have rebar3 and get the dependencies from the internet. And enterprises using this as well. However I think we shouldn't take away the option to build them in offline environments (or build systems that are not connected to the internet/use system dependencies).

Only if we already have a handful of Erlang applications that a distro wants to ship. It would be nice if a user can just install them via their system package manager. Have their config files manages by their package manager and use all their regular workflow. And not learn Erlang specific tools (if they are no Erlang developer). And for this reason distributions should be able to package Erlang applications properly :)

[weiss]

@tsloughter, FWIW, I'm on the Erlang side of the fence. I ran into the issue you mentioned: Some distros split up OTP, this breaks things for my users, they submit issues in my trackers, I'm annoyed. I don't use distro packages for Erlang development. I'm super-happy with Rebar3 in general and its dependency handling specifically.

However, I think this isn't the topic at hand. I work on software that isn't just used in-house but offered to the public, such as ejabberd, and we're switching to Rebar3 for building it. Distribution packaging is a great way of making such software available, and it frees upstream projects from some of the burdens involved with publishing software (it fills distro trackers with tickets!). Therefore, I'm thankful that package maintainers such as @jubalh and @debalance are putting so much work into this, they're doing an awesome job. So, for me personally, it actually is a priority to make their life easier, or at least not unnecessarily hard, when packaging Erlang software.

[tsloughter]

@weiss thanks, this actually brings up what I thought was the reason for these packages and with @jubalh and @debalance here maybe I can learn more and we figure this out.

I always wondered why ejabberd wasn't packaged as a whole release -- except for optional dependencies -- instead of being broken up? It is interesting to hear openSUSE doesn't break up Erlang like Debian does. I wonder then if when ejabberd is installed on openSUSE does it install all of Erlang/OTP in order to run ejabberd, instead of just the applications ejabberd needs and would have if it was packaged as an OTP release?

To me needing to package applications for running programs like ejabberd is different from wanting to package them and install for development purposes.

Is it not allowed to build an application with rebar3 handling the deps and then packaging its resulting application, _build/prod/lib/<app>? Meaning rebar3 needs to be able to build with global apps just so a distro can build the apps based on its own packages instead of rebar3 packages even though the resulting contents of _build/prod/lib/<app> is the same?

@jubalh you mention not taking away the option to build offline, I agree and we are working on that. But that won't necessarily be the same as working off of any global system packages since we intended to still respect the locks and its checksums of hex packages. So it will still have the issue of, you can easily build offline the contents of _build/prod/lib/<app>, just not necessarily with an openSUSE installed package of each dep.

I suppose what I describe might end up being cargo vendor which you mention is ugly for this :(. I'm going to look at cargo vendor today.

[ferd]

One of the issues is that we allow people to override and specify how some dependencies are built, and track on which compiler version they were built (so we can rebuild on the current version when it happens). This is simple to do when the dependencies are dependencies, and ignored when they're part of a standard OTP distribution (or in ERL_LIBS).

The challenge though is that when rebar3 gets told "the deps include libraries a, b, and c", well we go and build a, b, and c. The idea of having a switch to tell rebar3 "ignore the {deps, ...} config and lock files" is sort of foreign because my reaction to it would be: just remove it from the config file and don't tell us to do two things at once (build the deps then don't).

If the dep isn't specified but is available in ERL_LIBS, then I believe rebar3 will ignore them, but relx and other things that load libraries dynamically will find them fine.

IMO given the way rebar3 works (all providers are sort of independent and are free to interpret the config however they want), the simplest way to make rebar3 not worry about a dep is to not give it a configuration that demands to fetch a dep. So taking a config file, stripping the {deps, ...} tuple, and say saving it patched up as rebar.libs.config and passing in REBAR_CONFIG=rebar.libs.config ought to work by just giving it a separate config file to be used just for that free-standing build, without vendoring (assuming the libs are in ERL_LIBS).

Lock files may still be a problem though, but I think they're doing their job by warning you (or preventing you) from building an inconsistent project where the source and version of dependencies isn't as specified originally.

[tsloughter]

Right, and not just relx (rebar3 release) will work, compilation will still work as well when the deps are removed from rebar3 config/lock if they are in the system libraries (or anything under $ERL_LIBS). The deps are just to tell rebar3 what to build, the fact a module includes a header file and thus must have that application available at compile time works whether that header file is in a dep handled by rebar3 or is pre-built and installed globally.

[jubalh]

IMO given the way rebar3 works (all providers are sort of independent and are free to interpret the config however they want), the simplest way to make rebar3 not worry about a dep is to not give it a configuration that demands to fetch a dep. So taking a config file, stripping the {deps, ...} tuple, and say saving it patched up as rebar.libs.config and passing in REBAR_CONFIG=rebar.libs.config ought to work by just giving it a separate config file to be used just for that free-standing build, without vendoring (assuming the libs are in ERL_LIBS).

Understood. And I guess what we ask for is to give us an option to make rebar3 ignore the deps tuple and lock file. That's it :)

So not each distribution has to patch each Erlang package that they are building.

[debalance]

If the dep isn't specified but is available in ERL_LIBS, then I believe rebar3 will ignore them, but relx and other things that load libraries dynamically will find them fine.

IMO given the way rebar3 works (all providers are sort of independent and are free to interpret the config however they want), the simplest way to make rebar3 not worry about a dep is to not give it a configuration that demands to fetch a dep. So taking a config file, stripping the {deps, ...} tuple, and say saving it patched up as rebar.libs.config and passing in REBAR_CONFIG=rebar.libs.config ought to work by just giving it a separate config file to be used just for that free-standing build, without vendoring (assuming the libs are in ERL_LIBS).

That's what I have been doing for many years with packages that build using rebar 2. For some reasons I have long forgotten, the skip_deps=true switch was insufficient. I don't mind patching rebar.config/script/lock to stop it from fetching+building deps on its own, as long as there is a simple way for it to find and use the pre-built deps (from other distribution packages) in global system locations like /usr/lib/erlang/.

BTW, if you want the full Erlang/OTP installation on Debian, you can just install / depend on erlang instead of erlang-<whatsoever>. Debian tries to support as many different systems/architectures as possible, so by enabling folks to only install what they really need we allow for installations with less disk space.

[debalance]

I always wondered why ejabberd wasn't packaged as a whole release -- except for optional dependencies -- instead of being broken up? It is interesting to hear openSUSE doesn't break up Erlang like Debian does. I wonder then if when ejabberd is installed on openSUSE does it install all of Erlang/OTP in order to run ejabberd, instead of just the applications ejabberd needs and would have if it was packaged as an OTP release?

For one Debian usually does not allow embedded code copies of libraries, which arguably many parts of ejabberd singled out into their own git repositories have become. The most important reason though is to be able to fix security issues easily without having to rebuild and replace a huge package. E.g. in the past I had to fix erlang-p1-tls and could do so without touching ejabberd itself. The positive effects would be way greater though if there were more applications than just ejabberd using that library.

[tsloughter]

For some reasons I have long forgotten, the skip_deps=true switch was insufficient.

What was insufficient about skip_deps=true?

BTW, if you want the full Erlang/OTP installation on Debian, you can just install / depend on erlang instead of erlang-<whatsoever>. Debian tries to support as many different systems/architectures as possible, so by enabling folks to only install what they really need we allow for installations with less disk space.

Hm, surprised to hear installing the erlang package is supposed to work. I assume that is what users have been doing when they then encounter issues with certain OTP apps not working. I'll have to look into that more at some point soon.

[jubalh]

I saw that the old rebar just had SKIP_DEPS=true as mentioned in this thread as well. Is there any particular reason why such a solution wasn't implemented in rebar3?

Actually all that distros really need is not much: The best way for us would be if we could just get SKIP_DEPS=true in rebar3.

Would be great if this could be done!

[ferd]

This isn't obvious to make work, because the model of rebar3 itself just isn't that focused on this anymore. The compiler task knows it requires dependencies. Anything that depends on the compiler then transitively relies on the deps having been fetched. What we did for Elixir's toolchain was implement a 'bare compiler' where you give it all the paths of the dependencies and it just builds the .beam files. That compiler is a standalone provider that does not depend on dependencies, and on which no other task really depends either.

This surfaces a few problems of the approach:

• there are also sometimes plugins required to make builds work, and those aren't picked up from ERL_LIBS or anything else and are fetched for the project. Elixir projects sometimes report failures on these to this day
• building the Erlang files themselves is insufficient for things such as releases, which require a broader set of configurations and depend on the normal 'compile' task.
• some projects only build successfully when they can manage to apply overrides to dependencies, which can't be done if they're system libs, so builds that work in isolation may start failing when trying to set them up for a package manager -- their working depends on re-compiling with a project-local scope.

Are these acceptable tradeoffs? Because the current model for the application just isn't necessarily friendly. So "skip deps" on compilation also means there's a second change set needed for "skip deps on releases", which is also a distinct one required for "skip deps on bundling the release as a tarball" because each of these tasks declares dependencies that transitively relies on having installed deps. The switch has to be defined either at the config level (drop the deps and lockfile tuple) or as an environment variable (such that the deps fetching can return even if it has found nothing, but who knows what that then means when trying to run dialyzer or building escripts!)

I'm pointing this out because the actual mechanism you're looking for is not necessarily one that skips deps, but one that substitutes a functionality that represents a basic assumption of project structure for a distinct one that's useful in the context of a package manager only. This substitution in turn may break any other rebar3 task that shared that original assumption.

So the amount of success of skipping deps is contextual. Maybe you're good skipping deps and just getting beam files. Maybe there's a need to support a release. Are escripts also required? Do dialyzer runs count? Are projects with complex plugins desired or we're good with just a subset of projects that we're looking to package? I'm assuming we have no intent of running tests and test dependencies, but maybe some build processes do? What about documentation building?

We could have a simple "skip the deps" and make sure the blast radius is restricted in cases of compiling (and probably releases), but extending this to work for all the other commands (and their tests) is quite a demanding task. Scoping is important.

[weiss]

I totally get the potential issues with Rebar3 plugins, and with overrides.

What I don't quite understand is: If some dependency (say, elli) is installed into the global code:lib_dir() and removed from Rebar3's deps list / lock file. Is there any difference to any other application shipped with OTP (say, ssl), from the Rebar3 provider's point of view? Wouldn't compile, release, escript, dialyzer, ct, and so on use that dependency just fine (assuming there's no plugins/overrides issues in the way)?

[ferd]

In that case there shouldn't; tools like escript and whatnot will avoid re-bundling these libraries. Dialyzer will likely have to rebuild their PLT cache files each time the libraries' makeup changes.

I expect Relx to be fine as well. You'll get odd behaviour whenever you end up with two distinct applications depending on the same library but at two different versions. Those can live together at the same time in the root lib_dir (so long as the directory is named like appname-x.y.z), but the VM will automatically select the newest version each time unless the release specification is accurate about which version it requires (it likely isn't if it has been built with project-local requirements). Providers like escript or dialyzer will default to the newest version.

This means that you will likely need to undertake the task of managing all versions installed and of clamping down on all desirable paths for releases, or hope that the newest installed is correct across the whole environment for other tasks. This is likely to be challenging.

[jubalh]

This means that you will likely need to undertake the task of managing all versions installed and of clamping down on all desirable paths for releases, or hope that the newest installed is correct across the whole environment for other tasks. This is likely to be challenging.

I think rebar3 shouldn't take care of this in the case when the dependency is installed globally. Just take what's there and if they are incompatible it's the distros job to fix this. We do this in other areas as well.

[tsloughter]

@weiss right, everything should work fine. The deps are just to tell rebar3 what it needs to fetch and build, removing a dep from there but having it in the system libs will work like any system library.

oh wait, Fred mentioned rebar3 bare compile again and I realized we never got an answer if that works for your uses @jubalh and @debalance ?

It is basically skip_deps and is how mix tells rebar3 to ignore deps.

[debalance]

For some reasons I have long forgotten, the skip_deps=true switch was insufficient.

What was insufficient about skip_deps=true?

It is insufficient and build fails:

build log

``` rebar compile skip_deps=true -vv DEBUG: Evaluating config script "/build/erlang-p1-tls-1.1.13/rebar.config.script" DEBUG: Consult config file "/build/erlang-p1-tls-1.1.13/rebar.config" DEBUG: Rebar location: "/usr/bin/rebar" DEBUG: Consult config file "/build/erlang-p1-tls-1.1.13/src/fast_tls.app.src" DEBUG: is_app_available, looking for App p1_utils with Path "/build/erlang-p1-tls-1.1.13/deps/p1_utils" DEBUG: Directory expected to be an app dir, but it doesn't exist (yet?): /build/erlang-p1-tls-1.1.13/deps/p1_utils DEBUG: Available deps: [] DEBUG: Missing deps : [{dep,"/build/erlang-p1-tls-1.1.13/deps/p1_utils", p1_utils,".*", {git,"https://github.com/processone/p1_utils.git", {tag,"1.0.23"}}, false}] DEBUG: Plugins requested while processing /build/erlang-p1-tls-1.1.13: [] DEBUG: Predirs: [] ==> erlang-p1-tls-1.1.13 (compile) DEBUG: Matched required ERTS version: 12.3.1 -> .* DEBUG: Matched required OTP release: 24 -> .* DEBUG: Min OTP version unconfigured DEBUG: is_app_available, looking for App p1_utils with Path "/build/erlang-p1-tls-1.1.13/deps/p1_utils" DEBUG: Directory expected to be an app dir, but it doesn't exist (yet?): /build/erlang-p1-tls-1.1.13/deps/p1_utils Dependency not available: p1_utils-.* ({git, "https://github.com/processone/p1_utils.git", {tag,"1.0.23"}}) ERROR: compile failed while processing /build/erlang-p1-tls-1.1.13: rebar_abort make[1]: *** [/usr/share/dh-rebar/make/dh-rebar.Makefile:126: rebar_compile] Error 1 dh_auto_build: error: make --no-print-directory -f /usr/share/dh-rebar/make/dh-rebar.Makefile build returned exit code 2 make: *** [debian/rules:11: binary] Error 25 dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2 ```

Until just now I was not aware of REBAR_DEPS_PREFER_LIBS="TRUE", exporting that in the Makefile fixes the build with skip_deps=true, that is great news! :+1:

Regarding rebar3 I haven't had time yet to try making a package with rebar3 bare compile.

[weiss]

Fred mentioned rebar3 bare compile again and I realized we never got an answer if that works for your uses

I guess there's cases where distro packages would want to use rebar3 release …

I guess what we ask for is to give us an option to make rebar3 ignore the deps tuple and lock file. That's it :smile: So not each distribution has to patch each Erlang package that they are building.

That's how I understood it, yes. @ferd and @tsloughter mentioned various potential gotchas you might run into. I think those are good points, so the question is indeed:

Are these acceptable tradeoffs?

Which is obviously to be decided by you, the Rebar3 maintainers, as you'd end up having to support such functionality.

My humble opinion would still be that yes, the tradeoffs are acceptable. I'd assume a SKIP_DEPS knob would do the trick for most projects in practice. As for the remaining ones, distro maintainers are used to running into issues when packaging software. Erlang packagers will probably learn how to cope with common pitfalls, and might ask us for help with others. But that happens either way. I.e., I'd assume a SKIP_DEPS knob wouldn't really add support load on top, which would be my main criterion as a maintainer. I'd clearly document such a knob as "may break certain uses, use at own risk", or just not document it at all, and call it a day :smile:

[jubalh]

@ferd @tsloughter did our (distro maintainer) perspective reach you? I think the gist of it would just be: please give us a (hidden?) SKIP_DEPS :)

@ferd btw I'm reading your book right now. It's cool :)

[tsloughter]

@jubalh was still waiting to hear if rebar3 bare compile works since it is "skip deps" used by mix.

[jubalh]

I think that would work for libraries, but what about projects? I would need rebar3 release there, right?