Homebrew / brew

🍺 The missing package manager for macOS (or Linux)
https://brew.sh
BSD 2-Clause "Simplified" License
41k stars 9.62k forks source link

`brew install` painfully slow if Homebrew not up to date #6490

Closed djanowski closed 5 years ago

djanowski commented 5 years ago

Follow up on https://github.com/Homebrew/brew/issues/4755.

What you were trying to do (and why)

What: Use Homebrew to install dependencies in a CI environment (CircleCI/macOS in this case).

We're using the CircleCI image for XCode 10.3.0: https://circle-macos-docs.s3.amazonaws.com/image-manifest/build-903/index.html

Why: Because our build scripts need tools not included in macOS, and Homebrew is the same tool we use in our local environments.

What happened (include command output)

Homebrew takes 2-3 minutes to complete a brew install.

Command output

$ /usr/bin/time brew install --verbose --debug jq
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 3 taps (homebrew/core, homebrew/cask and caskroom/versions).
==> New Formulae
aws-cdk
base91
benthos
berglas
bitwise
broot
clp
codespell
coinutils
cpu_features
dive
fastmod
faust
ghc@8.6
git-revise
go-md2man
go@1.12
grpc-swift
kubebuilder
lazydocker
lerna
libsigc++@2
lightgbm
llvm@8
minizip2
msgpack-tools
now-cli
nushell
octant
open-image-denoise
pastel
pdfcpu
pdftk-java
pokerstove
protobuf@3.7
pulp
pygments
ripgrep-all
scc
scs
starship
wiiuse
wtfutil
zrepl
==> Updated Formulae
abcmidi
abyss
ace
ack
activemq
activemq-cpp
adwaita-icon-theme
afflib
afl-fuzz
agedu
aircrack-ng
algernon
aliyun-cli
allure
alpine
amap
amazon-ecs-cli
ammonite-repl
amqp-cpp
amtk
angband
angular-cli
anjuta
annie
ansible
ansible-lint
ansible@1.9
ansible@2.0
ansiweather
ant
anycable-go
apache-arrow
apache-flink
apache-geode
apache-spark
apib
appscale-tools
apr-util
aqbanking
arangodb
aravis
armadillo
arpack
arping
asdf
asio
ask-cli
aspell
astrometry-net
atk
atkmm
atlantis
atomist-cli
auditbeat
autorest
avfs
avro-c
avro-cpp
avro-tools
aws-google-auth
aws-okta
aws-sdk-cpp
awscli
axel
azure-cli
azure-storage-cpp
babel
babl
bacula-fd
badtouch
balena-cli
ballerina
baobab
bareos-client
basex
bash
bash-snippets
bat
bazel
bazelisk
bbftp-client
bcftools
bear
bedtools
bee
berkeley-db
bibtexconv
bigloo
binaryen
bind
biogeme
bison
bit
bitchx
bitcoin
bitrise
blueutil
boost
boost-bcp
boost-build
boost-mpi
boost-python
boost-python3
botan
breezy
bro
btfs
btparse
btpd
buildifier
buildkit
buku
bundletool
burp
c-blosc
cabal-install
cadaver
caddy
caf
caffe
cairomm
calicoctl
camlp4
camlp5
cargo-completion
cash-cli
catch2
ccache
ceres-solver
certbot
certstrap
cfn-lint
cfr-decompiler
cgdb
cglm
cgrep
chakra
chapel
charm-tools
checkbashisms
checkstyle
chronograf
cimg
circleci
citus
cjdns
ckan
clamav
clingo
clojure
closure-compiler
cmake
cmix
cmus
cnats
cockroach
cointop
collectd
collector-sidecar
commandbox
compcert
composer
conan
configen
conjure-up
consul
consul-template
contentful-cli
convox
coturn
cpl
cppcheck
cpprestsdk
cql
credstash
cromwell
crowdin
cryptol
crystal
crystal-icr
csound
cubelib
cucumber-cpp
curl
curl-openssl
cypher-shell
cython
dartsim
datamash
dav1d
davix
davmail
dbmate
dcmtk
dcos-cli
ddgr
deno
dependency-check
desktop-file-utils
devspace
dhall
dhall-bash
dhall-json
dialog
diamond
dieharder
diff-pdf
diff-so-fancy
diffoscope
diskus
distcc
dita-ot
django-completion
dmd
dmg2img
dnscrypt-proxy
dnscrypt-wrapper
dnsperf
dnsviz
docfx
docker
docker-completion
docker-compose
docker-machine
docker-machine-completion
doctl
docutils
doitlive
dosbox-x
dovecot
doxygen
drafter
drone-cli
dtc
dub
duck
duo_unix
duplicity
dyld-headers
dylibbundler
dynare
eccodes
efl
ejabberd
ekg2
elasticsearch
elektra
elinks
elm-format
emacs
embree
embulk
emscripten
encfs
enchant
entr
envconsul
epic5
eralchemy
erlang
erlang@20
erlang@21
eslint
etcd
ethereum
ettercap
exempi
exiftool
exim
exiv2
expat
exploitdb
faas-cli
fabric
fastjar
faudio
fauna-shell
fb-client
fd
fdroidserver
feedgnuplot
feh
ffmpeg
ffmpeg@2.8
ffsend
fibjs
file-roller
findutils
fio
firebase-cli
flac
flintrock
flow
flowgrind
fluid-synth
fluxctl
flyway
fn
folly
fontforge
fonttools
fossil
fpp
fq
fragroute
freeipmi
freeradius-server
freetds
frpc
frps
frugal
fselect
fstrm
fuse-emulator
fuse-zip
futhark
gambit-scheme
gammu
gatsby-cli
gauge
gcc
gdal
gdcm
gdk-pixbuf
gdl
gdub
geant4
gearman
geckodriver
gedit
geoipupdate
gerbil-scheme
get_iplayer
getdns
getmail
getxbook
ghc
ghr
gifski
git
git-annex
git-cola
git-crypt
git-extras
git-quick-stats
git-secret
git-series
gitbucket
gitg
gitlab-runner
gitleaks
gitmoji
gitversion
gjs
gkrellm
glances
glib
glib-networking
glib-openssl
glibmm
globus-toolkit
glooctl
gloox
gmic
gmsh
gmt
gnunet
gnupg
gnupg-pkcs11-scd
gnuradio
gnutls
go
go-jira
go@1.11
gobby
gobject-introspection
gocryptfs
godep
golang-migrate
goofys
gopass
goreleaser
gpac
gphoto2
gpsim
gradle
grafana
grakn
graph-tool
graphene
graphicsmagick
grok
groonga
groovy
grpc
grpcurl
gsettings-desktop-schemas
gsl
gsmartcontrol
gsoap
gspell
gst-plugins-bad
gst-plugins-good
gstreamermm
gtk+3
gtk-doc
gtk-vnc
gtkmm
gtkmm3
gtksourceview4
gtksourceviewmm
gtksourceviewmm3
gtmess
gtranslator
gwenhywfar
gwyddion
h2o
h3
hadolint
hapi-fhir-cli
haproxy
harfbuzz
hashcat
hashpump
hbase
heimdal
helmfile
helmsman
help2man
hfstospell
highlight
hive
hledger
hlint
homebank
htpdate
http_load
httpd
httperf
httpie
httping
httrack
hub
hugo
hydra
hyperfine
hyperkit
hyperscan
i2p
i2pd
i386-elf-gcc
icarus-verilog
icecast
icecream
ike-scan
imagemagick
imagemagick@6
imageoptim-cli
imap-uw
imapfilter
inlets
innotop
insect
interactive-rebase-tool
intercal
ios-sim
ios-webkit-debug-proxy
iperf
iperf3
ipfs
ipmitool
ipopt
ipv6calc
ipython
ircd-hybrid
ircii
irssi
ispc
istioctl
isync
janet
jdupes
jemalloc
jenkins
jenkins-lts
jfrog-cli-go
jhipster
jing-trang
jinja2-cli
john-jumbo
joplin
jose
jpeg-turbo
jruby
jsonnet
jsonrpc-glib
juise
juju
jvgrep
k3d
k6
kafka
kafkacat
kahip
kakoune
kibana
kim-api
knot
kops
kore
kotlin
krb5
kube-aws
kubecfg
kubectx
kubernetes-cli
kubernetes-helm
kubernetes-service-catalog-client
kubeseal
kubespy
kustomize
lablgtk
lammps
lasso
lastpass-cli
launchdns
lazygit
ldapvi
ldc
ldid
ldns
lean-cli
ledger
lego
lftp
lgogdownloader
libarchive
libbitcoin
libbitcoin-blockchain
libbitcoin-client
libbitcoin-consensus
libbitcoin-database
libbitcoin-explorer
libbitcoin-network
libbitcoin-node
libbitcoin-protocol
libbitcoin-server
libbluray
libcouchbase
libdap
libdazzle
libdvbpsi
libevent
libevhtp
libewf
libexosip
libfaketime
libfreefare
libftdi
libgcrypt
libgda
libgee
libgig
libgit2
libglademm
libgnomecanvasmm
libheif
libimagequant
libimobiledevice
libiodbc
libiscsi
libjwt
liblcf
libmaa
libmatio
libmodbus
libmowgli
libmxml
libmypaint
liboauth
libogg
libomp
libopendkim
libopenmpt
libosinfo
libosip
libosmium
libp11
libpeas
libphonenumber
libpq
libpqxx
libpulsar
libqalculate
libraw
librdkafka
libre
librealsense
librsvg
librsync
libsecret
libsigc++
libsignal-protocol-c
libslax
libsoup
libspng
libssh
libssh2
libstrophe
libtasn1
libtins
libtorrent-rasterbar
libtrace
libu2f-host
libu2f-server
libusb
libuv
libvirt
libvisio
libvmaf
libvoikko
libvterm
libwebsockets
libxkbcommon
libxml++
libxml++3
libxmlsec1
libzdb
lighttpd
linkerd
links
liquibase
liquid-dsp
lldpd
llvm
lmod
logrotate
logstash
logtalk
lolcat
lsd
lsof
luvit
lxc
lynx
lz4
mackup
mafft
mage
magic-wormhole
mailutils
makensis
makepkg
mame
mariadb
mariadb-connector-c
mariadb-connector-odbc
mariadb@10.0
mariadb@10.1
mariadb@10.2
mariadb@10.3
maven
mawk
maxwell
mbedtls
md5sha1sum
mdds
media-info
mediaconch
megacmd
megatools
memcached
memcacheq
menhir
mercurial
mesa
meson
metabase
metricbeat
mfterm
mgba
micronaut
midnight-commander
mill
miller
mingw-w64
minio
minio-mc
miniserve
mit-scheme
mitmproxy
mkcert
mkl-dnn
mktorrent
mkvtoolnix
mmark
mmseqs2
moarvm
mockserver
moco
modules
moe
molecule
monero
monetdb
mongo-c-driver
mongoose
monit
monitoring-plugins
monkeysphere
mono
mosh
mosquitto
mp3cat
mpg123
mrboom
msitools
mtr
mupdf
mutt
mycli
mydumper
mysql
mysql-client
mysql-connector-c++
mysql@5.7
mytop
n
nagios
nagios-plugins
nano
nativefier
nats-server
nats-streaming-server
ncdc
ncdu
ncmpcpp
nco
ncrack
ncview
neo4j
neofetch
neomutt
neon
neovim
net-snmp
netdata
netlify-cli
newman
newsboat
nfdump
nghttp2
nginx
ngircd
nickle
nim
nlohmann-json
nmap
nmh
nnn
node
node-build
node@10
node@8
nomad
nopoll
noti
nqp
nrpe
nsd
nspr
nsq
nss
ntl
ntp
nuget
numpy
nwchem
nylon
nyx
nzbget
oath-toolkit
ocaml
ocaml-findlib
ocaml-num
ocamlbuild
ocamlsdl
ocproxy
ocrmypdf
octave
ode
offlineimap
omega
omniorb
oniguruma
ooniprobe
opa
open-mpi
openapi-generator
openblas
opencolorio
openconnect
opencv
opencv@2
opencv@3
openfortivpn
openimageio
openkim-models
openldap
openrct2
opensaml
opensc
openssh
openssl
openssl@1.1
openvpn
operator-sdk
ophcrack
opusfile
orc
orc-tools
ori
osc
oscats
osquery
osslsigncode
owfs
p11-kit
p7zip
packer
packer-completion
pagmo
paket
pandoc-citeproc
pango
pangomm
parallel
passenger
pc6001vx
pcb2gcode
pdfcrack
pdftoipe
pdns
pdnsrec
pegtl
percona-server
percona-toolkit
percona-xtrabackup
petsc
petsc-complex
pev
pgbouncer
pgcli
pgroonga
pgweb
phoronix-test-suite
php
php-cs-fixer
php@7.1
php@7.2
phpmd
phpstan
phpunit
picard-tools
pijul
pilosa
pioneer
pip-completion
pipx
pjproject
pkcs11-helper
planck
plantuml
platformio
pmd
png++
pngquant
poco
podofo
pony-stable
ponyc
poppler
postgis
postgresql
postgresql@10
postgresql@9.4
postgresql@9.5
postgresql@9.6
postgrest
pre-commit
prefixsuffix
presto
procyon-decompiler
profanity
proj
prometheus
proteinortho
protobuf
protobuf-c
protobuf-swift
protobuf@3.6
proxytunnel
pspg
psqlodbc
pulledpork
pulseaudio
pulumi
pure-ftpd
pushpin
puzzles
pwntools
pwsafe
pybind11
pyenv
pygobject3
pypy
pypy3
pyqt
pyside
python
python@2
pyvim
qalculate-gtk
qca
qemu
qmmp
qpdf
qpid-proton
qrupdate
qscintilla2
qsoas
qt
quantlib
quickjs
r
rabbitmq-c
rakudo
range-v3
rc
rclone
rdesktop
re-flex
re2
re2c
readline
rebar3
redo
redsocks
remctl
rename
reop
repo
reposurgeon
riff
ripgrep
rke
rmlint
robot-framework
rom-tools
root
roswell
rpm
rsyslog
rtags
rtmpdump
ruby
ruby-build
ruby@2.4
ruby@2.5
rust
rustup-init
s-lang
s-nail
s2geometry
s3-backer
s3ql
sagittarius-scheme
saldl
salt
sane-backends
saxon
sbcl
sblim-sfcc
sbt
scala
scala@2.12
scalapack
scamper
sccache
scdoc
schismtracker
scipy
scons
scrcpy
scrypt
scw
sdhash
sdl2
serf
serverless
shadowenv
shadowsocks-libev
shairport
shairport-sync
shared-mime-info
shellcheck
shellharden
shellinabox
shibboleth-sp
ship
shogun
siege
signal-cli
sile
silk
simh
simple-scan
singular
sip
sipsak
siril
sk
skaffold
skinny
skopeo
sleuthkit
slowhttptest
slrn
smali
sn0int
snakemake
snapcraft
sngrep
snort
snownews
socat
sofia-sip
softhsm
solarus
solr
sonarqube
sonarqube-lts
sonobuoy
source-highlight
sourcekitten
sourcery
spdlog
spdylay
speedtest-cli
sphinx
sphinx-doc
spice-gtk
spim
spin
spiped
spirv-cross
sqlcipher
sqlmap
squid
src
srt
ssh-audit
ssh-permit-a38
sshfs
ssldump
sslsplit
sstp-client
step
stolon
stone-soup
storm
stow
streamlink
stress-ng
strongswan
stunnel
stuntman
subnetcalc
subversion
subversion@1.8
suite-sparse
sundials
supervisor
svtplay-dl
swagger-codegen
swagger-codegen@2
swi-prolog
swiftformat
swiftlint
swig
sylpheed
sync_gateway
syncthing
synfig
sysbench
sysdig
talloc
tarantool
tarsnap
taskell
tcl-tk
tcpdump
tcpflow
tdlib
tealdeer
tectonic
telegraf
telegram-cli
teleport
template-glib
termius
terraform
terraform-inventory
terraformer
terragrunt
terrahub
tesseract
testdisk
tflint
thrift
tika
tile38
tinc
tintin
tiny-fugue
tinyxml2
tmate
tmux
tomcat
tomcat-native
tomcat@7
tomcat@8
tomee-plus
tomee-webprofile
topgit
topgrade
tor
traefik
trafficserver
transcrypt
translate-shell
triton
ttyd
tunnel
tvnamer
twarc
twemcache
twine-pypi
typescript
u-boot-tools
ucloud
udunits
unbound
ungit
unrar
unrtf
unshield
unzip
upscaledb
uptimed
urh
uwsgi
v8
vala
valabind
valgrind
vapoursynth
vapoursynth-imwri
vapoursynth-ocr
vapoursynth-sub
varnish
vault
vault-cli
vaulted
velero
verilator
vert.x
vgmstream
vifm
vim
vim@7.4
vimpc
vips
virgil
visp
vit
vitetris
vnstat
vte3
vtk
vulkan-headers
wandio
watchexec
watchman
wcslib
wdc
webdis
webpack
webtorrent-cli
weechat
whois
widelands
wine
winetricks
wireguard-go
wireguard-tools
wiremock-standalone
wireshark
wla-dx
wp-cli
wskdeploy
wtf
wxmaxima
x11vnc
x265
xapian
xcodegen
xdot
xmake
xml-security-c
xml-tooling-c
xmount
xmrig
xonsh
xrootd
xsimd
xtensor
xxhash
yamllint
yank
yaws
yelp-tools
ykman
yosys
you-get
youtube-dl
z3
zabbix
zbackup
zboy
zelda-roth-se
zim
zimg
zint
znc
zsdx
zstd
zsxd
==> Renamed Formulae
confluent-oss -> confluent-platform
transmission -> transmission-cli
==> Deleted Formulae
cclive
cherokee
csup
ctunnel
engine_pkcs11
frag_find
freetds@0.91
ftimes
httest
irods
konoha
kontena
liblacewing
libopkele
llvm@3.9
llvm@5
mongodb
mongodb@3.0
mongodb@3.2
mongodb@3.4
mongodb@3.6
mysql-cluster
nordugrid-arc
packetbeat
percona-server-mongodb
pincaster
postgres-xc
skipfish
taisei
thc-pptp-bruter
tlsdate
tn5250
voms
xar-mackyle

/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/jq.rb
Warning: jq 1.6 is already installed and up-to-date
To reinstall 1.6, run `brew reinstall jq`
      199.32 real       141.30 user        59.66 sys
  

What you expected to happen

Homebrew can complete installing a simple formula in a reasonable amount of time. 30 seconds?

Note that I'm installing an already-installed dependency, so this time doesn't include actually installing the program.

Step-by-step reproduction instructions (by running brew commands)

More information

HOMEBREW_VERSION: 2.1.8
ORIGIN: https://github.com/Homebrew/brew
HEAD: 549a40bedb32171912a449d8d0e7bcae02d5b432
Last commit: 9 weeks ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 888b778d0290c07b77e9796a5be8c72066e05986
Core tap last commit: 9 weeks ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_NO_ANALYTICS_THIS_RUN: 1
CPU: quad-core 64-bit ivybridge
Homebrew Ruby: 2.3.7 => /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby
Clang: 10.0 build 1001
Git: 2.22.0 => /usr/local/bin/git
Curl: 7.54.0 => /usr/bin/curl
Java: 1.8.0_222
macOS: 10.14.4-x86_64
CLT: 10.3.0.0.1.1562985497
Xcode: 10.3 => /Applications/Xcode-10.3.app/Contents/Developer
CLT headers: 10.3.0.0.1.1562985497

Running with HOMEBREW_NO_ANALYTICS=1 doesn't have any noticeable impact.

If I run brew update --verbose --debug --force, the bulk of the time seems to be spent looping over all taps:

...
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FromPathLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/flyway.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaContentsLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/flyway.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FromPathLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/fn.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaContentsLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/fn.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FromPathLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/folly.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaContentsLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/folly.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FromPathLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/fontforge.rb
...
close-issue-app[bot] commented 5 years ago

Whoops, it looks like you created an issue without filling out the checklist and providing all the needed information from one of the issue templates: https://github.com/Homebrew/brew/issues/new/choose

As we need all this information to help you we're closing this issue. If you add the necessary information to this issue (don't create a new or duplicate issue, please) and comment to let us know, a Homebrew maintainer will check it out when they get the chance. If it's deemed to be a valid issue then this issue may be reopened.

Sorry!

djanowski commented 5 years ago
MikeMcQuaid commented 5 years ago

Follow up on #4755.

From that issue:

To be clear, I'm open to this being improved but until we've got something actionable here I don't think it's worth having an open issue.

  • SSH into a CircleCI XCode 10.3 image

This is an issue with CircleCI not keeping their images reasonable up-to-date, not with Homebrew.

If you want to avoid updating: there's documentation for doing that in man brew (although we won't fix any issues you encounter with that setup). Otherwise: we'll accept pull requests to make this faster but I'd be surprised if there's anything that's going to dramatically improve performance here.