Feature Request: Preventing formulae updates from breaking their dependencies

[ahundt]

Key problems

1. Package A is updated from 1.0 to 2.0, breaking package B which depends on A 1.0.
   • breakage is not always detected
     • Example: ceres depends on an exact version of eigen (a simple bottle refresh could fix it)
   • breakage can't always be fixed
     • B may not support A 2.0 until its next release a month from now
2. Package B,C depend on different versions of package A. Package D depends on both B, C and sometimes A directly. (other variations on this theme exist)
3. Options sometimes break invisibly
4. Cross tap dependencies may change

   Solutions with most backing

These are reasonably easy, well liked ideas that will mitigate (but may not completely solve) the above

• [ ] allow a limited set of versions into homebrew supported taps with an aliasing mechanism for the latest version so dependent libraries can have a soft landing until they support new versions
• [ ] define criteria for accepting requests/proposals to make certain options recommended by default

  proposed solutions

This is the full list of proposals (not definite upcoming features). Check marks indicate they are implemented

• [x] checking all formulae for broken dylib linkage
• [x] rearranged brew audit output format
• [ ] allowing versions for widely used directly into homebrew supported taps where appropriate
  • this seems particularly popular and supported by several commenters
• [ ] "maximal options" test
• [ ] dependencies on specific versions in formulas
• [ ] allowing version phase-ins / phase-outs
• [ ] side by side install of versions (where possible)
• [ ] keg only installs of versions
• [ ] alias the latest version i.e. boost is boost160
• [ ] allow multiple acceptable dependency versions to be specified perhaps with a preferred option
• [ ] easy way to remap dependencies like brew install mypackage --override-dep boost ahundt/boost155. could be a normal brew command or perhaps an easy way to change the formula files?
• [ ] move libraries and system services (not end-user applications or GUIs) that have dependencies in the Homebrew org to being versioned on the next major upgrade
• [ ] allow multiple versions to be specified in a single formula
• [ ] define criteria where proposals to make certain options recommended by default is encouraged

  posed problems without solutions

these are from the ongoing conversation beyond the initial request posed above

• [ ] version dependency conflicts for multiple packages. Here is an example:

A: depends on boost (latest)
B: depends on boost155 (newest feasible version)
C: depends on A, B, sometimes various boost versions

• [ ] cross tap dependencies may change

  Original Feature Request

I keep having libraries I depend on get broken because those dependencies' own dependencies are changed out from under them. In particular, this happens most frequently when flags other than the default are needed.

Two examples: https://github.com/Homebrew/homebrew-science/issues/3454 https://github.com/Homebrew/homebrew-science/pull/3512

Perhaps there is a way to avoid these problems that also meets the crucial criteria of the homebrew team being willing to put it into practice? :-)

Thanks for your thoughts and considering my request!

[MikeMcQuaid]

Package B,C depend on different versions of package A. Package D depends on both B, C and sometimes A. (other variations on this theme exist)

I don't actually think this is a problem; it's something that blocks package D from being packaged in pretty much any package manager and is a problem with their development process.

Otherwise: yeh, I agree 👍

[CamJN]

I just stumbled across this issue and I have been frustrated with version incompatibilities before (reverting QT5.6 to 5.5 for the capybara-webkit gem was annoying and isn't going to be fixed so now qt5 has to be pinned forever, and I can never build anything that requires QT5.6+ at build time without remembering to install the new version and then switch the links back when I'm done so that the next time I bundle update everything's not broken again).

However I'd rather deal with that migraine than this bit: For mysql I see us suffixing the binary names and data directories but probably not e.g. ports. Breaking the data directory for a database across versions? Where does the data actually live? Is the data migrated forward with each upgrade? Even major upgrades? But what if something depends on the old db version (the whole point), do you have to create a cluster from all versions and sync the data constantly? Because that often isn't possible across DB versions (every database I've ever used has done this at one point or another; opentsdb/influx/mysql/postgres just recently). If you want to avoid a huge support nightmare I'd steer away from that idea hard. It's already hard enough to talk to the postgres/mysql people because homebrew's not a blessed/supported config so they can just point at that and not help.

I think there's a lot of good that could come from versioning some libraries but please please please don't do it to the databases.

[MikeMcQuaid]

I think there's a lot of good that could come from versioning some libraries but please please please don't do it to the databases.

We will version databases in some form, just not sure what that will be yet. As a counterpoint under the current system if you have MySQL 5.6 data and upgrade to 5.7 temporarily then getting your data 5.6 compatible again is a non-trivial process.

[ahundt]

Package B,C depend on different versions of package A. Package D depends on both B, C and sometimes A. (other variations on this theme exist)

I don't actually think this is a problem; it's something that blocks package D from being packaged in pretty much any package manager and is a problem with their development process.

You're right. Thinking about it more, I guess I mostly run into this when adding new dependencies which supply new functionality to software I'm developing myself, or when porting software from another OS or package management system like ubuntu. I guess I was just thinking of "things that would be nice to avoid" and not "things a package manager can actually do anything about", haha.

[geoff-nixon]

Thinking about it more, I guess I mostly run into this when adding new dependencies which supply new functionality to software I'm developing myself ...

I'm probably going to catch some shit for even mentioning this, but for several years now I've been using a personal fork of homebrew which does not use dynamic linkage at all. Its a pretty simple hack, just replacing every instance of "./configure" with ./configure --disable-shared --enable-static --with-pic for autotools-based formulae, and adding -DBUILD_SHARED_LIBS=0 to "cmake" for cmake-based based formulae.

This (obviously) takes care of this issue (and most other linkage headaches) for 95% of the formulae I use, and (in most likelihood) at least half of all formulae in general. For the rest I use --interactive to eliminate dynamic linkage manually.

There are so many reasons to do this, the most important of which, for me, as I'm developing my own software, is to avoid things like this; but when developing an app/program that will be distributed in binary form, I actually don't really understand how anyone would be able to use homebrew without doing something like this.

[MikeMcQuaid]

@geoff-codes The opposite: I'm pretty interested in learning more about that and what it would take to make your fork a bit easier.

[geoff-nixon]

Oh!

In the past I've had several unpleasant run-ins with some homebrew maintainers who vociferously advocated for dynamically linking as much as possible (that issue I linked above doesn't address this exact idea per se, but I trust you can gather my meaning). I have thus largely avoided proposing anything related to what is now the "brew" repo, for some time now. I have even avoided putting anything in a public fork: my worry was someone might come across it, adopt it, and suddenly a few weeks I'd be maintaining a fork to which the upstream maintainers were diametrically opposed.

But I know there's been a significant "changing of the guard" since that period, so, if you're serious:

---

There are actually several "side-advantages" of doing this as well, namely,

• Many, many formulae that are now either keg-only or in -dupes no longer need be so, because there is no issue any with library paths or such nonsense, nor worries about ABI compatibility with system libraries. I actually usually intentionally shadow some system libraries. For example, I can (and do) use the newer, and significantly faster whilst also being able to target older SDKs (I usually build against 10.7). And I even have fully-linked, non-sequestered libiconv installed, believe it or not 😈.

I get faster code and great compatibility across systems. All it costs me is a few kb's in my executables.

• It also makes a homebrew install almost completely portable between directories and systems. I don't having to worry about instal_name_tool -changeing thousands of executables and libraries. I have an expanded brew script that basically:
  1. Caches its own location, and checks to make sure it hasn't been moved since the last invocation.
  2. If it detects a move, it 'grep-and-sed's any references to the location that exist in text files (we're basically only talking about .pc files, then reinstalls openssl, which is one a small handful of formulae that hardcode their absolute paths within the binaries themselves.
• It also makes cross-platform compatibility significantly less painful. I use the same basic principal (and a lot of the same code) with a very modified linuxbrew that produces binaries and libraries that are largely interchangeable between distros.

Presently, the code I have is pretty crude, and mostly consists of shell scripts that copy formulae to a temporary directory, 'grep-and-sed's as appropriate; then installs the formula using a file:// URI.

So, if indeed this is something you'd want to consider experimenting with, I'd be happy to clean it all up and implement it in Ruby for something you could pull into a topic branch...?

[sjackman]

with a very modified linuxbrew that produces binaries and libraries that are largely interchangeable between distros.

Linuxbrew bottles can run on any distro. The bottles target glibc 2.19 and gcc 4.8. If the host's versions are older than this, it seamlessly installs the Linuxbrew glibc and gcc, which takes about five minutes, since gcc is bottled. I think your goal however is to distribute your compiled executables outside of Linuxbrew?

[sjackman]

As an alternative to static linking, you can copy all the shared executables that an executable depends upon into its libexec directory and put that directory first in its RPATH. I prefer that solution to static linking myself.

[sjackman]

If the executable is installed in …/bin and its libraries in …/libexec and you set the RPATH to $ORIGIN/../libexec, the executable is relocatable without needing to fix up the executable using patchelf.

[scpeters]

@geoff-codes the static linking idea sounds to me a bit like Ubuntu snappy, which I think is great for leaf applications. If you want to build a framework to use in other applications, my impression is that static linking would be less helpful. I wonder what percentage of homebrew installs are for leaf applications vs. frameworks.

[geoff-nixon]

@sjackman

I think your goal however is to distribute your compiled executables outside of Linuxbrew?

It's both. That is, one goal is to be able to compile executables that can be distributed completely outside/without brew, while still being able to use brew handle building libraries, dependencies, as well as building utilities, toolchain, etc.

As an alternative to static linking, you can copy all the shared executables that an executable depends upon into its libexec directory and put that directory first in its RPATH. I prefer that solution to static linking myself.

Yeah, I'm just going to have to disagree with you there.

@scpeters Yes, I agree its a bit like snappy, but it is exactly sta.li.

In fact, their FAQ makes all the points I would make on this argument. And its highly recommended reading to anyone interested, no matter which side of the aisle you're on on this topic.

[MikeMcQuaid]

Closing in favour of https://github.com/Homebrew/brew/issues/620. Thanks for all the discussion, folks.