[homebrew] check for existing packages first

[NickLaMuro]

When the brew $package_manager is detected and used for ruby-install, it will attempt two things when installing dependencies:

• $ brew install $@
• $ brew upgrade $@

The first will not install packages that already exist (will installs ones that do not), and will raise an error code if any packages attempting to be installed are already installed. The second will be triggered on a non-zero exit code of the first and upgrade all packages and dependencies of those packages that need it.

While on the surface, this seems okay since newer dependencies are usually a good thing, in practice this usually leads to "bricking" all other older ruby installations since readline is usually upgraded as a result. Which in a dev environment, is probably not necessary, and leads to the following when attempting to use irb:

$ irb
~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require':
    dlopen(~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/x86_64-darwin15/readline.bundle, 9):
    Library not loaded: /usr/local/opt/readline/lib/libreadline.7.dylib (LoadError)
  Referenced from: ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/x86_64-darwin15/readline.bundle
  Reason: image not found - ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/x86_64-darwin15/readline.bundle
        from ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/irb/ext/save-history.rb:12:in `<top (required)>'
        from ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/irb/extend-command.rb:243:in `save_history='
        from ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/irb/context.rb:91:in `initialize'
        from ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/irb.rb:426:in `new'
        from ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/irb.rb:426:in `initialize'
        from ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/irb.rb:383:in `new'
        from ~/.rubies/ruby-2.3.3/lib/ruby/2.3.0/irb.rb:383:in `start'
        from ~/.rubies/ruby-2.3.3/bin/irb:11:in `<main>'

After a re-install (for the example above: $ ruby-install ruby-2.3.3), running irb them greets you with the following:

$ irb
Ignoring bcrypt-3.1.12 because its extensions are not built.  Try: gem pristine bcrypt --version 3.1.12
Ignoring bindex-0.5.0 because its extensions are not built.  Try: gem pristine bindex --version 0.5.0
Ignoring bootsnap-1.1.2 because its extensions are not built.  Try: gem pristine bootsnap --version 1.1.2
Ignoring byebug-10.0.2 because its extensions are not built.  Try: gem pristine byebug --version 10.0.2
Ignoring curses-1.0.2 because its extensions are not built.  Try: gem pristine curses --version 1.0.2
# ----  x45 other gems in my case ----
irb(main):001:0>

The Fix: Install missing packages only

Like the pacman package manager case below this, this code adds additional steps to determine the $missing_pkgs, and only attempts to install those. Unfortunately, brew lacks the functionality to accurately print out a list of missing formule or only install missing from a list, so the patch here does it manually.

Alternate approach (not taken)

An alternative command form of the $ brew list command exists:

$ brew list --versions

which will list the passed in packages (along with their respective version numbers) to STDOUT. An undocumented part of this feature is that any missing packages that are not included in the output result the command in returning an non-zero exit code, causing the previous commands to continue as they did.

The diff ends up being a one line change:

 brew)
   local brew_owner="$(/usr/bin/stat -f %Su "$(command -v brew)")"
+  sudo -u "$brew_owner" brew list --versions "$@" ||
   sudo -u "$brew_owner" brew install "$@" ||
   sudo -u "$brew_owner" brew upgrade "$@" || return $?

Unfortunately, this is an all or nothing approach, that when one of the packages is missing, the previous functionality would be still triggered for every package passed in. The $missing_pkgs approach was favored over this approach because it was more granular, and almost entirely prevents the issue mentioned above.

[kbrock]

thanks nick. thorough as usual

[postmodern]

Does homebrew not have a way to install packages, but omit the ones which are already installed? apt install and dnf install both omit already installed packages. Surely there must be a way to offload this check to brew?

[NickLaMuro]

This is a bit of a old PR at this point, and my guess homebrew has changed a bit, so it is possible it has changed since then (for the better, or worse in this regard).

Specifically regarding dependencies, the linked PR above:

https://github.com/Homebrew/brew/issues/6103

Has shown to me that homebrew will "un-apologetically" upgrade packages for you, and pinning packages to avoid this is basically unsupported.

I don't think I still have the output for this any more, but in recent history, I upgraded or installed a package that had something like a OpenSSL or python dependency, and ended up upgrading 30+ of my other installs because those also had dependencies on that upgraded package...

---

Point being, I can look into this more, but I am skeptical based on past experience that there is going to be a good solution for this.

That, or I am truly ignorant to how homebrew works after using it for all of these years.

[postmodern]

Ideally, I'd like to not upgrade the packages, if I don't have to. If the homebrew packages do get upgraded and you do have issues with gems which contain C bindings, you can re-compile those gems by running gem pristin --extensions and avoid having to bundle update your apps.

[kbrock]

it seems openssl and readline are the main two culprets, though I have seen it in other places.

Do we still want the upgrade? the list was already pruned.

Thanks for the gem pristine --extensions command. I knew that years ago but have since forgotten it.

[kbrock]

It looks like ruby-build now pulls out the openssl and packages the library with each version of ruby that is built. Not sure if this is a viable option here.

Not thrilled about having an ssl version that is not upgraded - but it removes a constant stress locally. I think this has been an issue since ruby 1.9.2. (1.8.4 had a different issue)

[jrochkind]

I don't think I still have the output for this any more, but in recent history, I upgraded or installed a package that had something like a OpenSSL or python dependency, and ended up upgrading 30+ of my other installs because those also had dependencies on that upgraded package...

I don't understand what is going on at all, but this is what's happening for me now when I try ruby-install ruby 2.7.2 on MacOS. Ends up with brew trying to upgrade dozens of packages, including python (how is python a ruby dependency? It is not, but something weird is going on). Takes a long time if you let it do everything, and possibly messes up my system. ruby-install 0.8.1.

Found this issue searching for anyone mentioning this.

I think it maybe is indeed openssl related, like if you're trying to change the version of openssl installed on the system, brew is trying to helpfully re-build everything that depends on openssl, which is, like, everything. maybe?

It is... inconvenient to say the least.

[kbrock]

@jrochkind You are not alone here.

A number of people are unhappy with the way homebrew behaves. Lets be clear, the issue you are seeing is actually part of homebrew and not part of ruby-install.

Nick put in this PR to avoid the changes in homebrew, but the correct solution may be to configure homebrew to not behave badly in the first place.

The homebrew team said this change happened in 2016, but it seems most people started feeling this much more recently than that.

check out HOMEBREW_NO_AUTO_UPDATE=1 at man page

best of luck

[jrochkind]

Thanks. I don't think HOMEBREW_NO_AUTO_UPDATE is a solution.

That just effects whether brew update is called automatically before doing other homebrew commands. While this can be annoying cause homebrew update can take a while... that's not the problem we are talking about here. Even if not called automatically, from the days before it was, you still had to call homebrew update manually sometimes.

The problem is the way homebrew is deciding to reinstall much of my system when i use ruby-install to install a new ruby. That is not the brew update operation on it's own doing that, it doesn't do that when you just call brew update. I think maybe (and I'm not sure I understand it) because ruby-install wants a new/different version of openssl, and installing this (at least as the "default" openssl in homebrew?) triggers a reinstall of anything that depends on openssl? Because, maybe, otherwise those things would be broken, because they are linked against an openssl that isn't there anymore? I may be wrong about that diagnosis, I really don't understand what's going on.

Even if never ever calling homebrew update would prevent htis from happening (not sure?), that's obviously not realistic, I am going to run homebrew update eventually anyway, even if I do it manually because HOMEBREW_NO_AUTO_UPDATE is turned on.

I can believe that the problem is caused by some bad design in homebrew (I don't know enough to have an opinion), but I don't think there's going to be any simple solution on homebrew end, I think ruby-install is going to have to work around it, if it's going to be usable.

[havenwood]

@jrochkind For what it's worth, an immediate fix from the ruby-install side is to --no-install-deps and install any missing deps yourself. An option from the Homebrew side is to pin each dep, brew pin automake bison gdbm libffi libyaml openssl readline style.

[NickLaMuro]

@havenwood Thanks for the review, and sorry for the delay in getting back. I hope to take some time to rebase this PR and implement the suggestions you have given, but I probably won't get around to it until next week.

Regardless, again, thanks for looking at this! It is very appreciated.