Dependent library version: single or list/ranges?

[UweRaabe]

There can be problems when multiple libraries depend on the same library, or the project depends on a library that also other libraries depend on. Example: libA and libB both depend on libX. Both libA and libB specify a version of libX they require. What if these versions don't match? Can libA specify: I need libX with version "2.x" or "at least 2.1" or "2.x to 4.x"? Which version should be selected when there is a choice? If libA is allowed to only only 1 version of libX, do we need multiple versions of libA for every version of libX even when nothing in libA has to be changed? This is not bound to source libraries only, but can also hold for binary ones.

[vincentparrett]

I'm still trying to get my head around version compatibility. If the source code is available then provided they are actually api compatible, it's simple. The complexity comes in when a package doesn't provide the source. In that case, the version compatibility rules will need to be strict.

As for resolving dependency conflicts, the most compatible option will have to be determined. There will be dependency conflicts that simply cannot be resolved without updating the dependencies. To quote myself (in my rfc blog post under dependencies)

This is certainly more complicated than other platforms, and a significant amount of work to get right (ps, if you think it isn't, you haven't considered all the angles!)

I guess the important thing is that when a dependency cannot be resolved, we need to report why and make suggestions on how to fix it (ie, taking updates if that will fix it). Nuget drives me mad sometimes with it's shitty error messages around dependency conflicts.

[UweRaabe]

There actually is a decent flexibility even with non-source packages. There are cases when a compiled DCU still works even when a dependent unit is changed and recompiled. Otherwise it wouldn't work to just copy f.i. Vcl.ComCtrls.pas into your project folder, tweak it somehow and compile the project. There are about a dozen Vcl units depending on Vcl.ComCtrls. IIRC, this works unless the interface is changed (which can also happen when compiler options are changed).

The good thing is that these dependencies can be tested in advance. If I declare that a dcu-only package libA version 1.3 is compatible with a source only package libX version 1.0 to 1.4 I am able (and obliged) to test that before it finds its way into the package manager. If that weren't possible there would be no way for a binary package to depend on a source only package in the first place.

[vincentparrett]

DCU compatibility is a fickle thing. The best answer I have seen was from Allen Bauer a few years ago when I asked on G+

Ok, this is an interesting subject and one that many folks seem to think is voodoo and magic. The rules are deceptively simple...

Any material change to an existing interfaced symbol, directly or indirectly, will render that dcu incompatible with prebuilt units that depend on this unit.

Let's break this down a little more. What is a "material" change?

1. Anything that could change the layout of a structured type (object, class, record, etc).
2. A change in symbols within a structured scope. This means even the addition of a non-virtual method to a base class is a "material" change.
3. A change in method or global function signature.

Remember, I also said indirectly... For instance, a change to a type used as the parameter of a given global function will also cause a compatibility error. Even if that type is the same size and semantic.

The key points here are changes to existing symbols. What this means is that you can, many times, get away with adding another symbol to the unit without rendering it incompatible with existing pre-built dcu files. This is logical be cause there cannot be a reference to a symbol that didn't previously exist. However, this can also backfire, because you once new code references the new symbols, that new code cannot be used with the old version of the dcu. Logically because of the above "material change" rule. Deleting a symbol is a "material" change.

Even if you think you followed all those rules, there can still be cases where dcus can be rendered incompatible. Some of those are the result of a compiler change in a patch that fixes something critical but could render dcus built with the new version incompatible with older versions of the compiler. The newer, fixed version would usually be made resilient enough to load the older dcu files and compile against them, but new dcus written cannot be used with the older compiler. This is rare, but has happened a few times over the last couple of decades.

I think allowing open ended compatibility (e.g >= 1.1 ) would be too risky, as a dcu breaking change could be introduced at any time and the package author may not even realise they are breaking the compatibility.

So the moral of the story is - the only way to know for sure is to test and then specify specific versions as compatible. Of course if package authors adhere to Semver practices this could perhaps be relaxed a little.

[vincentparrett]

Oh, and a key point to take from Allen's post is the word "indirectly" - in a dependency chain

A < B < C

The author of B could take an update to A, make no other changes and this may still cause a dcu version change for B which breaks C. I'm not sure how a package author really test for this?

[UweRaabe]

I see your point - and it also holds true for source packages. Alas I fear the consequences of a single version dependency.

To avoid version deadlocks a package A should accept as much versions of all packages C, D, E it depends on. If this means a separate package A version for each combination of all C, D, E versions it can be compiled with, that can give a pretty high number. Now imagine that for a multi-level dependency chain.

If the hurdles to even test all these combined dependencies are that high, the package author will most likely opt for the easier way: Eliminate the dependency by including all depending code into the package with adjusted unit names (I have done exactly that with the Delphi AST units in my CodeCoverage plugin). Unfortunately this will not work if components are going to be registered in the IDE, which requires renaming those components, too.

[vincentparrett]

Does anyone understand the dcu format? I was thinking as part of the package creation we could create a hash of the dcu versions, that way we could discover the incompatibilities more easily. It was an idea I had a few years ago but never explored it further.

[UweRaabe]

There is a Delphi Compile Unit Parser: http://hmelnov.icc.ru/DCU/index.eng.html