Question about rebar.lock

[tothlac]

It's only a question related to what can be locked in rebar.lock.

We always commit our rebar.lock files because in most cases we use {branch, "master"} in our rebar.config files, so rebar.lock is where the version of dependencies are locked. This way our builds are reproducible, meaning if no code has been changed in a repository, the build on the repo should always have the same result.

Unfortunately, there are two exceptions when we can't lock something in rebar.lock:

• plugins
• test dependencies from the profiles section of rebar.config

For instance, some weeks ago the following happened: We did not lock the version of rebar3_lint plugin in rebar.config. There were some not backward-compatible changes in the plugin, so the build was successful, and after some minutes when the plugin changed we had lots of errors reported by lint.

I know locking plugins the same way as dependencies are locked would not be easy, but as I see test dependencies also can't be locked. Would it be possible to add locking mechanism at least for the dependencies declared in the profiles section?

[aboroska]

Plugins: As a workaround for plugins it is possible to request a specific version:

{plugins, [{rebar3_lint, "0.4.1"}]}.

[tsloughter]

What @aboroska mentions is also the case for test and other profiles, you can manually lock int he rebar.config, I know that is a pain when using a master branch, but it is the best solution at this time.

I can't recall if non-default profiles are as difficult (really impossible in the case of plugins because of how they are installed) to implement support for in the lock file. There should be old issues related to this that will shed light on the possibility. I'll try to dig around for them today.

[tothlac]

@aboroska thanks, that's how we fixed the issue. We use an older version of lint, because on one-day errors reported by the plugin started appearing in all of our repositories. That was not a surprise, the fix was easy, but this is still not the real locking algorithm implemented for dependencies.

@tsloughter I was only asking it. I do understand implementing it for plugins would be really difficult using the current implementation, but maybe it would be a nice feature for test dependencies.

[ferd]

Yeah it should be possible for other dependencies, the sort of reason for not doing it is that profiles can proliferate indefinitely. You can call rebar3 as test,prod,osx,linux ct which will merge the dep set of all 5 profiles (default is implicit), and when you want to upgrade, you'll need to also call rebar3 upgrade as <profiles> for each combination for it to change. And note that rebar3 as osx,test compile is not the same as rebar3 as test,osx compile -- those are two different dep sets that would yeld two lock sets because the profiles may cause clashing transitive dep priorities.

Currently the sort of upside is that if you rebar3 upgrade <app>, it implicitly upgrades it for all profiles because it's only locked for default.

I think back then when we saw the sort of combinatorial explosion of lockable profile sets and just decided not to go there. We could very well implement it, but the usability of it is sort of messy to figure out.

[ericmj]

Would it really be necessary to keep distinct locks for different profiles? I understand that you would want different dependencies for different profiles, but in the case where deps overlap is it necessary that those deps should not be locked to the same versions, is it even a good idea to have different versions of the same dep for different profiles?

I am asking this because I would really like to see a future where all production deps (plugin or otherwise) would be listed as the deps under the hex package. If we need to make changes to hex to make that possible I would happy to discuss that.

[ferd]

There's no other option based on the current model:

    A       B
   / \     / \  
  D   E   F   G
  |   |
  J   K
  |
  I1

This might be your default profile. When using a dependency for tests, you might end up in a situation such as:

    A       B       C1
   / \     / \     / \
  D   E   F   G   H   I2
  |   |
  J   K
  |
  I1

As per our build rules, I2 has to be chosen prior to I1. Adding dependencies in a profile necessarily can change the final order of applications that should be included. Changing these rules is a massive breaking change that I wouldn't enable lightly.

[ericmj]

Would it be possible to at least get default/prod plugins added to deps and locked?

[ferd]

Plugins are the hardest to add, because they may need to be built and compiled before we're even done fetching deps, since plugins can specify how to fetch more deps. They are in theory possible to implement, but in practice we've made things so flexible that the implementation would be scary hairy.

[ericmj]

Do you think there is a way forward to at least improve some things in this area? Maybe push some of the functionality plugins provide into deps so that plugins are needed less?

I think a common use case for libraries to use plugins is to add compilers, but compilers are only needed after deps are fetched so if we could make it possible to add rebar compilers from deps we wouldn't need plugins for this use case.

[tsloughter]

Only needed after deps are fetched if there isn't a plugin that uses the compiler ;)

[tsloughter]

Also plugins can be used for fetching dependencies, so another place they can't wait until after fetching :(

[ericmj]

Only needed after deps are fetched if there isn't a plugin that uses the compiler ;)

Wouldn't the plugin have the compiler as a dependency if it required it?

Also plugins can be used for fetching dependencies, so another place they can't wait until after fetching :(

Right, I understand we can't cover every scenario. I am looking to see if there is anything we can improve, I understand we can't change how plugins work completely. Plugins fetching deps isn't an issue for Hex packages.

[tsloughter]

@ericmj I just remembered another issue. A plugin and dep can have the same dependency but with different versions. This was motivated by issues that occurred with projects back in the rebar2 days when this wasn't the case.

I'd fear that attempting to simply cover scenarios where none of the issues exist (like a duplicate dep) would lead to something too fragile.

[ferd]

Plugin fetching deps isn't an issue for hex packages, but it has long been an issue to the stability of projects in Erlang where at some point you have a plugin that requires an outdated version of a library your project uses, but only for things like logging or internal data structures that aren't shared.

We have decided a while ago that we'd prefer to keep the dep split because we'd rather have a plugin use an outdated JSON transitive dependency than have it force you to update your kafka protocol version and database client in order to output test coverage (this is not necessarily a case that happened, but I recall things of this magnitude happening in the past). Keeping the dep set split has pulled some of the complexity into the build tool in order to avoid that sort of big bang change in user projects.

[ferd]

Oh and I forgot. Since plugins can be downloaded before deps, it could create real funny edge cases. Imagine the following scenario:

1. The project defines a third party plugin to fetch dependencies over subversion
2. the plugin's dependency requires a serialization library on version 1.2.3
3. rebar3 fetches and builds both the plugin and its serialization library
4. rebar3 sees dependencies for the main project, which is a database client and fetches it
5. by fetching the database client, rebar3 discovers a git dependency to the same serialization library, but on version 2.3.4

Now we enter a forking point. We can:

• bail out due to the conflict (the project is now broken)
• decide to keep the plugin's version and ignore the project's (the project is now broken)
• decide to keep the project's version 2.3.4 (and now on the next time we validate version dependencies, the project is broken because it can no longer validate or update its own deps because the plugin for that can't work anymore)
• keep the plugin and dependency versions separate (everything works)

Forcing plugins into deps comes with the implication that we can't let people define custom resource plugins fetched over different protocols anymore without also cascading new failure modes outlined above. It's something that we could possibly do in a greenfield scenario but not easily today with tons of projects in the wild.

[tsloughter]

And all this is to say that I very much want plugins to be deps that are part of the hex package. But them being implemented before hex and not in conjunction with it likely make that not possible until either rebar4 (though at this point I don't think there is any good solution to do it even with backwards incompatible changes) or some sort of hex support for plugin dependencies, to allow for deps of different versions on the same package.

[ferd]

it also needs to work for non-hex dependencies regardless.