Homebrew / brew

🍺 The missing package manager for macOS (or Linux)
https://brew.sh
BSD 2-Clause "Simplified" License
41.3k stars 9.71k forks source link

`brew tests` failure: mismatched caveat message #15644

Closed osalbahr closed 1 year ago

osalbahr commented 1 year ago

brew doctor output

==> Downloading https://formulae.brew.sh/api/formula.jws.json
######################################################################### 100.0%
Your system is ready to brew.

Verification

brew config output

HOMEBREW_VERSION: 4.0.26
ORIGIN: https://github.com/Homebrew/brew
HEAD: eff45ef570f265e226f14ce91da72d7a6e7d516a
Last commit: 10 days ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: cf3cb18796664100f963873bfd2d603d80491b74
Core tap last commit: 10 days ago
Core tap branch: master
Core tap JSON: 07 Jul 08:55 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.10 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby
CPU: quad-core 64-bit skylake
Clang: N/A
Git: 2.41.0 => /bin/git
Curl: 7.81.0 => /bin/curl
Kernel: Linux 6.3.8-200.fc38.x86_64 x86_64 GNU/Linux
OS: Unknown
Host glibc: 2.35
/usr/bin/gcc: 11.3.0
/usr/bin/ruby: N/A
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A

What were you trying to do (and why)?

I was running brew tests. Was just curious what it does, after seeing it mentioned under How to Open a Pull Request - Create your pull request from a new branch. I also wanted to use it as a way to test if any changes I make in the near future introduce breakage.

What happened (include all command output)?

Also mentioned in #15635.

The message brew tests expected is in https://github.com/Homebrew/brew/blob/master/Library/Homebrew/caveats.rb#L58.

[dracula@fedora ~]$ podman run -it homebrew/brew
linuxbrew@b5132d7be8bc:~$ brew tests
Warning: tests is a developer command, so Homebrew's
developer mode has been automatically turned on.
To turn developer mode off, run:
  brew developer off

Fetching gem metadata from https://rubygems.org/.......
Using concurrent-ruby 1.2.2
Using minitest 5.18.1
Using zeitwerk 2.6.8
Using ast 2.4.2
Using bundler 2.3.26
Using public_suffix 5.0.1
Using bindata 2.4.15
Using msgpack 1.7.1
Using diff-lcs 1.5.0
Using byebug 11.1.3
Using coderay 1.1.3
Using docile 1.4.0
Using regexp_parser 2.8.1
Using did_you_mean 1.6.3
Using json 2.6.3
Using hana 1.3.7
Using hpricot 0.8.6
Using uri_template 0.7.0
Using method_source 1.0.0
Using plist 3.7.0
Using rainbow 3.1.1
Using mustache 1.1.1
Using rdiscount 2.2.7.1
Using rack 3.0.8
Using rexml 3.2.5
Using sorbet-runtime 0.5.10461
Using parallel 1.23.0
Using ruby-progressbar 1.13.0
Using ruby-macho 3.0.0
Using simplecov-html 0.12.3
Using rspec-support 3.12.1
Fetching ruby-prof 1.4.3
Using unicode-display_width 2.4.2
Using simplecov_json_formatter 0.1.4
Fetching stackprof 0.2.25
Using warning 1.3.0
Using i18n 1.14.1
Using tzinfo 2.0.6
Using parser 3.2.2.1
Using addressable 2.8.4
Using bootsnap 1.16.0
Using elftools 1.2.0
Using ecma-re-validator 0.4.0
Using pry 0.14.2
Using ronn 0.7.3
Using rspec-sorbet 1.9.2
Using rspec-core 3.12.2
Using rspec-expectations 3.12.3
Using parallel_tests 3.13.0
Using rspec-mocks 3.12.5
Using simplecov 0.22.0
Using activesupport 6.1.7.3
Using patchelf 1.4.0
Using rubocop-ast 1.29.0
Using json_schemer 0.2.24
Using rspec-github 2.4.0
Using rspec-its 1.3.0
Using rspec-retry 0.6.2
Using rspec_junit_formatter 0.6.0
Using rspec 3.12.0
Using simplecov-cobertura 2.1.0
Using rubocop 1.50.2
Using rubocop-capybara 2.18.0
Using rubocop-performance 1.17.1
Using rubocop-rails 2.19.1
Using rubocop-sorbet 0.7.0
Using rubocop-rspec 2.20.0
Installing stackprof 0.2.25 with native extensions
Installing ruby-prof 1.4.3 with native extensions
Bundle complete! 36 Gemfile dependencies, 67 gems now installed.
Bundled gems are installed into `./.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle`
Randomized with seed 63640
4 processes for 353 specs, ~ 88 specs per process
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................*.**.*...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) Language::Python#site_packages gives a different location between PyPy and Python 2
     # Python is not installed.
     # ./test/language/python_spec.rb:14

  2) Language::Python#user_site_packages can determine user site packages location
     # Python is not installed.
     # ./test/language/python_spec.rb:27

  3) Language::Python#homebrew_site_packages returns the Homebrew site packages location
     # Python is not installed.
     # ./test/language/python_spec.rb:20

  4) Language::Python#major_minor_version returns a Version for Python 2
     # Python is not installed.
     # ./test/language/python_spec.rb:7

...............................................................***.***.**............................................

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) Language::Python::Virtualenv::Virtualenv#pip_install accepts an array
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:44

  2) Language::Python::Virtualenv::Virtualenv#pip_install accepts a Resource
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:58

  3) Language::Python::Virtualenv::Virtualenv#pip_install accepts a string
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:24

  4) Language::Python::Virtualenv::Virtualenv#pip_install accepts a multi-line strings
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:32

  5) Language::Python::Virtualenv::Virtualenv#pip_install works without build isolation
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:70

  6) Language::Python::Virtualenv::Virtualenv#create creates a venv
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:17

  7) Language::Python::Virtualenv::Virtualenv#pip_install_and_link can link manpages
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:108

  8) Language::Python::Virtualenv::Virtualenv#pip_install_and_link can link scripts
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:85

.........
Failures:

  1) Caveats#caveats when f.keg_only is not nil when joining different caveat types together adds the correct amount of new lines to the output
     Failure/Error: expect(caveats).to include("if you don't want/need a background service")

       expected "something else\n\nformula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230707-385-yizafq/prefix,\nbecause some reason.\n\nWarning: formula_name provides a service which can only be used on macOS or systemd!\nYou can manually execute the service instead with:\n  /tmp/homebrew-tests-20230707-385-yizafq/cellar/formula_name/1.0/bin/cmd\n" to include "if you don't want/need a background service"
       Diff:
       @@ -1,8 +1,15 @@
       -if you don't want/need a background service
       +something else
       +
       +formula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230707-385-yizafq/prefix,
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230707-385-yizafq/cellar/formula_name/1.0/bin/cmd

     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'
.........................................................................................................
.....................................................

Took 84 seconds (1:24)
Tests Failed
Running `brew update` twice beforehand makes the output a lot more verbose:
[dracula@fedora ~]$ podman run -it
Error: requires at least 1 arg(s), only received 0
[dracula@fedora ~]$ podman run -it homebrew/brew
linuxbrew@a99445ca3c8f:~$ brew update
==> Homebrew collects anonymous analytics.
Read the analytics documentation (and how to opt-out) here:
  https://docs.brew.sh/Analytics
No analytics have been recorded yet (nor will be during this `brew` run).

==> Homebrew is run entirely by unpaid volunteers. Please consider donating:
  https://github.com/Homebrew/brew#donations

Installing from the API is now the default behaviour!
You can save space and time by running:
  brew untap homebrew/core
Already up-to-date.
linuxbrew@a99445ca3c8f:~$ brew update               
Installing from the API is now the default behaviour!
You can save space and time by running:
  brew untap homebrew/core
Already up-to-date.
linuxbrew@a99445ca3c8f:~$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: You have an unnecessary local Core tap!
This can cause problems installing up-to-date formulae.
Please remove it by running:
 brew untap homebrew/core
linuxbrew@a99445ca3c8f:~$ brew untap homebrew/core
Untapping homebrew/core...
Untapped 3 commands and 6721 formulae (7,096 files, 473.5MB).
linuxbrew@a99445ca3c8f:~$ brew doctor
Your system is ready to brew.
linuxbrew@a99445ca3c8f:~$ brew update && brew update
Already up-to-date.
Already up-to-date.
linuxbrew@a99445ca3c8f:~$ brew doctor
Your system is ready to brew.
linuxbrew@a99445ca3c8f:~$ brew tests 
Warning: tests is a developer command, so Homebrew's
developer mode has been automatically turned on.
To turn developer mode off, run:
  brew developer off

Fetching gem metadata from https://rubygems.org/.......
Using concurrent-ruby 1.2.2
Using ast 2.4.2
Using bindata 2.4.15
Using minitest 5.18.1
Using zeitwerk 2.6.8
Using public_suffix 5.0.1
Using msgpack 1.7.1
Using coderay 1.1.3
Using byebug 11.1.3
Using bundler 2.3.26
Using docile 1.4.0
Using hpricot 0.8.6
Using json 2.6.3
Using did_you_mean 1.6.3
Using regexp_parser 2.8.1
Using hana 1.3.7
Fetching unf_ext 0.0.8.2
Using diff-lcs 1.5.0
Using method_source 1.0.0
Using mustache 1.1.1
Using parallel 1.23.0
Fetching racc 1.7.1
Using plist 3.7.0
Using rack 3.0.8
Using rainbow 3.1.1
Using rdiscount 2.2.7.1
Using rexml 3.2.5
Using rspec-support 3.12.1
Using sorbet-runtime 0.5.10461
Using ruby-progressbar 1.13.0
Using ruby-macho 3.0.0
Fetching ruby-prof 1.4.3
Using unicode-display_width 2.4.2
Using simplecov-html 0.12.3
Using simplecov_json_formatter 0.1.4
Fetching stackprof 0.2.25
Installing stackprof 0.2.25 with native extensions
Installing racc 1.7.1 with native extensions
Installing ruby-prof 1.4.3 with native extensions
Installing unf_ext 0.0.8.2 with native extensions
Using warning 1.3.0
Using i18n 1.14.1
Using tzinfo 2.0.6
Using elftools 1.2.0
Using addressable 2.8.4
Using bootsnap 1.16.0
Using pry 0.14.2
Using parallel_tests 3.13.0
Using ronn 0.7.3
Using rspec-core 3.12.2
Using rspec-expectations 3.12.3
Using rspec-mocks 3.12.5
Using rspec-sorbet 1.9.2
Using simplecov 0.22.0
Fetching unf 0.1.4
Installing unf 0.1.4
Fetching activesupport 6.1.7.4
Installing activesupport 6.1.7.4
Using patchelf 1.4.0
Using rspec-github 2.4.0
Using rspec-retry 0.6.2
Using rspec_junit_formatter 0.6.0
Using rspec-its 1.3.0
Using rspec 3.12.0
Using simplecov-cobertura 2.1.0
Fetching simpleidn 0.2.1
Installing simpleidn 0.2.1
Fetching json_schemer 1.0.3
Fetching parser 3.2.2.3
Installing json_schemer 1.0.3
Installing parser 3.2.2.3
Using rubocop-ast 1.29.0
Using rubocop 1.50.2
Using rubocop-capybara 2.18.0
Using rubocop-performance 1.17.1
Using rubocop-rails 2.19.1
Using rubocop-sorbet 0.7.0
Using rubocop-rspec 2.20.0
Bundle complete! 36 Gemfile dependencies, 69 gems now installed.
Bundled gems are installed into `./.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle`
Removing activesupport (6.1.7.3)
Removing ecma-re-validator (0.4.0)
Removing json_schemer (0.2.24)
Removing parser (3.2.2.1)
Removing uri_template (0.7.0)
Randomized with seed 1072
4 processes for 353 specs, ~ 88 specs per process
....................................................................................................................................................................................F..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F..................................................................................................................................................................................................................................................................................................................................................................................................................................F............................................................................................................................F..............................................................................................................................F.............................................F...................................................F..................................................................................F....F....................................................F..................................................................................F.......................................F....................................F...........................FFF.F..F.FF......F.................F....................F............................*..*.*.*.*..*.*.*..............................F.....................F.....F......................F..........F.....F..................F...............................F............................................................................................................................................................F..........****..............

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) Language::Python::Virtualenv::Virtualenv#pip_install accepts a multi-line strings
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:32

  2) Language::Python::Virtualenv::Virtualenv#pip_install accepts an array
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:44

  3) Language::Python::Virtualenv::Virtualenv#pip_install accepts a string
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:24

  4) Language::Python::Virtualenv::Virtualenv#pip_install accepts a Resource
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:58

  5) Language::Python::Virtualenv::Virtualenv#pip_install works without build isolation
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:70

  6) Language::Python::Virtualenv::Virtualenv#pip_install_and_link can link manpages
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:108

  7) Language::Python::Virtualenv::Virtualenv#pip_install_and_link can link scripts
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:85

  8) Language::Python::Virtualenv::Virtualenv#create creates a venv
     # Python is not installed.
     # ./test/language/python/virtualenv_spec.rb:17

  9) Language::Python#site_packages gives a different location between PyPy and Python 2
     # Python is not installed.
     # ./test/language/python_spec.rb:14

  10) Language::Python#major_minor_version returns a Version for Python 2
     # Python is not installed.
     # ./test/language/python_spec.rb:7

  11) Language::Python#user_site_packages can determine user site packages location
     # Python is not installed.
     # ./test/language/python_spec.rb:27

  12) Language::Python#homebrew_site_packages returns the Homebrew site packages location
     # Python is not installed.
     # ./test/language/python_spec.rb:20

Failures:

  1) Caveats#caveats when f.keg_only is not nil when joining different caveat types together adds the correct amount of new lines to the output
     Failure/Error: expect(caveats).to include("if you don't want/need a background service")

       expected "something else\n\nformula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230707-3751-82oa0l/prefix,\nbecause some reason.\n\nWarning: formula_name provides a service which can only be used on macOS or systemd!\nYou can manually execute the service instead with:\n  /tmp/homebrew-tests-20230707-3751-82oa0l/cellar/formula_name/1.0/bin/cmd\n" to include "if you don't want/need a background service"
       Diff:
       @@ -1,8 +1,15 @@
       -if you don't want/need a background service
       +something else
       +
       +formula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230707-3751-82oa0l/prefix,
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230707-3751-82oa0l/cellar/formula_name/1.0/bin/cmd

     # ./test/caveats_spec.rb:289:in `block (5 levels) in '
F...............................F.........................................F...........................................
.F..............................................................................F.............................................FFFFF..........FF.......................FFFFFFFFFFFFF..............FFF......FF....F.....................................................F.................................F......F.......F................................................................................................................................................................................................................................................................

Failures:

  1) RuboCop::Cop::FormulaAudit::ProvidedByMacos krb5 formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  2) RuboCop::Cop::FormulaAudit::ProvidedByMacos libedit formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  3) RuboCop::Cop::FormulaAudit::ProvidedByMacos libiconv formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  4) RuboCop::Cop::FormulaAudit::ProvidedByMacos apr formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  5) RuboCop::Cop::FormulaAudit::ProvidedByMacos berkeley-db formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  6) RuboCop::Cop::FormulaAudit::ProvidedByMacos cups formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  7) RuboCop::Cop::FormulaAudit::ProvidedByMacos ed formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  8) RuboCop::Cop::FormulaAudit::ProvidedByMacos libxslt formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  9) RuboCop::Cop::FormulaAudit::ProvidedByMacos curl formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  10) RuboCop::Cop::FormulaAudit::ProvidedByMacos zip formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  11) RuboCop::Cop::FormulaAudit::ProvidedByMacos ruby formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  12) RuboCop::Cop::FormulaAudit::ProvidedByMacos libxcrypt formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  13) RuboCop::Cop::FormulaAudit::ProvidedByMacos lsof formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  14) RuboCop::Cop::FormulaAudit::ProvidedByMacos tcl-tk formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  15) RuboCop::Cop::FormulaAudit::ProvidedByMacos zlib formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  16) RuboCop::Cop::FormulaAudit::ProvidedByMacos file-formula formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  17) RuboCop::Cop::FormulaAudit::ProvidedByMacos rpcgen formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  18) RuboCop::Cop::FormulaAudit::ProvidedByMacos cyrus-sasl formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  19) RuboCop::Cop::FormulaAudit::ProvidedByMacos gnu-getopt formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  20) RuboCop::Cop::FormulaAudit::ProvidedByMacos texinfo formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  21) RuboCop::Cop::FormulaAudit::ProvidedByMacos dyld-headers formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  22) RuboCop::Cop::FormulaAudit::ProvidedByMacos whois formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  23) RuboCop::Cop::FormulaAudit::ProvidedByMacos ncurses formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  24) RuboCop::Cop::FormulaAudit::ProvidedByMacos pod2man formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  25) RuboCop::Cop::FormulaAudit::ProvidedByMacos m4 formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  26) RuboCop::Cop::FormulaAudit::ProvidedByMacos bison formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  27) RuboCop::Cop::FormulaAudit::ProvidedByMacos bzip2 formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  28) RuboCop::Cop::FormulaAudit::ProvidedByMacos libarchive formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  29) RuboCop::Cop::FormulaAudit::ProvidedByMacos expat formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  30) RuboCop::Cop::FormulaAudit::ProvidedByMacos ncompress formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  31) RuboCop::Cop::FormulaAudit::ProvidedByMacos gperf formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  32) RuboCop::Cop::FormulaAudit::ProvidedByMacos pcsc-lite formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  33) RuboCop::Cop::FormulaAudit::ProvidedByMacos unzip formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  34) RuboCop::Cop::FormulaAudit::ProvidedByMacos llvm formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  35) RuboCop::Cop::FormulaAudit::ProvidedByMacos libpcap formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  36) RuboCop::Cop::FormulaAudit::ProvidedByMacos bc formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  37) RuboCop::Cop::FormulaAudit::ProvidedByMacos openldap formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  38) RuboCop::Cop::FormulaAudit::ProvidedByMacos icu4c formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  39) RuboCop::Cop::FormulaAudit::ProvidedByMacos net-snmp formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  40) RuboCop::Cop::FormulaAudit::ProvidedByMacos flex formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  41) RuboCop::Cop::FormulaAudit::ProvidedByMacos libressl formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  42) RuboCop::Cop::FormulaAudit::ProvidedByMacos swift formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  43) RuboCop::Cop::FormulaAudit::ProvidedByMacos unifdef formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  44) RuboCop::Cop::FormulaAudit::ProvidedByMacos libxml2 formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  45) RuboCop::Cop::FormulaAudit::ProvidedByMacos netcat formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  46) RuboCop::Cop::FormulaAudit::ProvidedByMacos sqlite formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  47) RuboCop::Cop::FormulaAudit::ProvidedByMacos ssh-copy-id formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  48) RuboCop::Cop::FormulaAudit::ProvidedByMacos pax formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  49) RuboCop::Cop::FormulaAudit::ProvidedByMacos libffi formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/provided_by_macos_spec.rb:42
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '
.......................................

Failures:

  1) brew --prefix prints the prefix for a Formula
     Failure/Error:
       expect { brew_sh "--prefix", "wget" }
         .to output("#{ENV.fetch("HOMEBREW_PREFIX")}/opt/wget\n").to_stdout
         .and not_to_output.to_stderr
         .and be_a_success

          expected block to output "/home/linuxbrew/.linuxbrew/opt/wget\n" to stdout, but output nothing

       ...and:

             expected block to not output to stderr, but output "Error: No available formula with the name \"wget\".\n"

          ...and:

             expected # to be a success
     # ./test/cmd/--prefix_spec.rb:16:in `block (2 levels) in '
     # ./test/support/helper/spec/shared_context/integration_test.rb:49:in `block (2 levels) in '

  2) RuboCop::Cop::FormulaAudit::UsesFromMacos xz formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  3) RuboCop::Cop::FormulaAudit::UsesFromMacos zsh formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  4) RuboCop::Cop::FormulaAudit::UsesFromMacos rsync formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  5) RuboCop::Cop::FormulaAudit::UsesFromMacos perl formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  6) RuboCop::Cop::FormulaAudit::UsesFromMacos openssl formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  7) RuboCop::Cop::FormulaAudit::UsesFromMacos vim formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  8) RuboCop::Cop::FormulaAudit::UsesFromMacos python formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  9) RuboCop::Cop::FormulaAudit::UsesFromMacos mandoc formula exists
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

       expected true
            got false
     Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
     # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  10) RuboCop::Cop::FormulaAudit::UsesFromMacos groff formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  11) RuboCop::Cop::FormulaAudit::UsesFromMacos cpio formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  12) RuboCop::Cop::FormulaAudit::UsesFromMacos expect formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  13) RuboCop::Cop::FormulaAudit::UsesFromMacos php formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  14) RuboCop::Cop::FormulaAudit::UsesFromMacos git formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  15) RuboCop::Cop::FormulaAudit::UsesFromMacos bash formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  16) RuboCop::Cop::FormulaAudit::UsesFromMacos gzip formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

  17) RuboCop::Cop::FormulaAudit::UsesFromMacos less formula exists
      Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }

        expected true
             got false
      Shared Example Group: "formulae exist" called from ./test/rubocops/uses_from_macos_spec.rb:20
      # ./test/support/helper/spec/shared_examples/formulae_exist.rb:9:in `block (3 levels) in '

Took 107 seconds (1:47)
Tests Failed

What did you expect to happen?

Tests Passed!

Step-by-step reproduction instructions (by running brew commands)

podman run -it homebrew/brew
brew tests
osalbahr commented 1 year ago

I found where the "actual" message comes from: https://github.com/search?q=repo%3AHomebrew%2Fbrew%20%22if%20you%20don%27t%20want%2Fneed%20a%20background%20service%22&type=code

I am not sure how to figure out which message is the correct one. I ran brew tests --verbose to get more outptut for that failure:

[master a8b51f3] commit
 1 file changed, 1 insertion(+), 1 deletion(-)
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 4 (delta 1), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (4/4), 307 bytes | 307.00 KiB/s, done.
From /tmp/homebrew-tests-20230707-98411-my7s72/prefix/Library/Taps/homebrew/homebrew-foo
   1d5f201..a8b51f3  master     -> origin/master

Failures:

  1) Caveats#caveats when f.keg_only is not nil when joining different caveat types together adds the correct amount of new lines to the output
     Failure/Error: expect(caveats).to include("if you don't want/need a background service")

       expected "something else\n\nformula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230707-98410-gukbci/prefix,\nbecause some reason.\n\nWarning: formula_name provides a service which can only be used on macOS or systemd!\nYou can manually execute the service instead with:\n  /tmp/homebrew-tests-20230707-98410-gukbci/cellar/formula_name/1.0/bin/cmd\n" to include "if you don't want/need a background service"
       Diff:
       @@ -1,8 +1,15 @@
       -if you don't want/need a background service
       +something else
       +
       +formula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230707-98410-gukbci/prefix,
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230707-98410-gukbci/cellar/formula_name/1.0/bin/cmd

     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'
HEAD is now at a8b51f3 commit

Is there a way to run only a specific test from brew tests?

carlocab commented 1 year ago

Is there a way to run only a specific test from brew tests?

From brew help tests:

❯ brew help tests
Usage: brew tests [options]

Run Homebrew's unit and integration tests.

[snip]

      --only                       Run only test_script_spec.rb. Appending
                                   :line_number will start at a specific
                                   line

You only get file-level granularity of test selection, though.

osalbahr commented 1 year ago

I discovered that the test passes on my macOS:

dracula@REDACTED ~ % brew tests --only caveats
Randomized with seed 58845
1 process for 1 spec, ~ 1 spec per process
...........................

Took 3 seconds
dracula@REDACTED ~ % brew config
HOMEBREW_VERSION: 4.0.28-67-ge9ac36a
ORIGIN: https://github.com/Homebrew/brew
HEAD: e9ac36a5424f97ec48d21aba31128576c16ffab2
Last commit: 3 days ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: b0d1abf200ff129aa8ec1a0dae1a13e3950b3116
Core tap last commit: 3 days ago
Core tap branch: master
Core tap JSON: 14 Jul 21:53 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: quad-core 64-bit kabylake
Clang: 14.0.3 build 1403
Git: 2.39.2 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 7.88.1 => /usr/bin/curl
macOS: 13.4.1-x86_64
CLT: 14.3.1.0.1.1683849156
Xcode: N/A

It also passes passes in a podman run --rm -it homebrew/brew on my macOS if I rn sudo apt-get update && sudo apt-get install build-essential beforehand:

linuxbrew@56816a3e97f7:~$ brew tests --only caveats
Randomized with seed 60684
1 process for 1 spec, ~ 1 spec per process
.........F............

Failures:

  1) Caveats#caveats when f.keg_only is not nil when joining different caveat types together adds the correct amount of new lines to the output
     Failure/Error: expect(caveats).to include("if you don't want/need a background service")

       expected "something else\n\nformula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230717-1166-1w37jco/prefix,\nbecause some reason.\n\nWarning: formula_name provides a service which can only be used on macOS or systemd!\nYou can manually execute the service instead with:\n  /tmp/homebrew-tests-20230717-1166-1w37jco/cellar/formula_name/1.0/bin/cmd\n" to include "if you don't want/need a background service"
       Diff:
       @@ -1,8 +1,15 @@
       -if you don't want/need a background service
       +something else
       +
       +formula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230717-1166-1w37jco/prefix,
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230717-1166-1w37jco/cellar/formula_name/1.0/bin/cmd

     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'

Took 3 seconds
Tests Failed
linuxbrew@56816a3e97f7:~$ sudo apt-get -y install build-essential

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  dirmngr dpkg-dev fakeroot gnupg gnupg-l10n gnupg-utils gpg-wks-client
  gpg-wks-server gpgsm libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libdpkg-perl libfakeroot libfile-fcntllock-perl
  libksba8 liblocale-gettext-perl lto-disabled-list xz-utils
Suggested packages:
  dbus-user-session libpam-systemd pinentry-gnome3 tor debian-keyring
  parcimonie xloadimage bzr
The following NEW packages will be installed:
  build-essential dirmngr dpkg-dev fakeroot gnupg gnupg-l10n gnupg-utils
  gpg-wks-client gpg-wks-server gpgsm libalgorithm-diff-perl
  libalgorithm-diff-xs-perl libalgorithm-merge-perl libdpkg-perl libfakeroot
  libfile-fcntllock-perl libksba8 liblocale-gettext-perl lto-disabled-list
  xz-utils
0 upgraded, 20 newly installed, 0 to remove and 2 not upgraded.
Need to get 2876 kB of archives.
After this operation, 9752 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblocale-gettext-perl amd64 1.07-4build3 [17.1 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 xz-utils amd64 5.2.5-2ubuntu1 [84.8 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libdpkg-perl all 1.21.1ubuntu2.2 [237 kB]
Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 lto-disabled-list all 24 [12.5 kB]
Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 dpkg-dev all 1.21.1ubuntu2.2 [922 kB]
Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 build-essential amd64 12.9ubuntu3 [4744 B]
Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libksba8 amd64 1.6.0-2ubuntu0.2 [119 kB]
Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 dirmngr amd64 2.2.27-3ubuntu2.1 [293 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfakeroot amd64 1.28-1ubuntu1 [31.5 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 fakeroot amd64 1.28-1ubuntu1 [60.4 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg-l10n all 2.2.27-3ubuntu2.1 [54.4 kB]
Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg-utils amd64 2.2.27-3ubuntu2.1 [308 kB]
Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-wks-client amd64 2.2.27-3ubuntu2.1 [62.7 kB]
Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-wks-server amd64 2.2.27-3ubuntu2.1 [57.5 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpgsm amd64 2.2.27-3ubuntu2.1 [197 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg all 2.2.27-3ubuntu2.1 [315 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libalgorithm-diff-perl all 1.201-1 [41.8 kB]
Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 libalgorithm-diff-xs-perl amd64 0.04-6build3 [11.9 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 libalgorithm-merge-perl all 0.08-3 [12.0 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfile-fcntllock-perl amd64 0.22-3build7 [33.9 kB]
Fetched 2876 kB in 1s (2074 kB/s)             
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package liblocale-gettext-perl.
(Reading database ... 14587 files and directories currently installed.)
Preparing to unpack .../00-liblocale-gettext-perl_1.07-4build3_amd64.deb ...
Unpacking liblocale-gettext-perl (1.07-4build3) ...
Selecting previously unselected package xz-utils.
Preparing to unpack .../01-xz-utils_5.2.5-2ubuntu1_amd64.deb ...
Unpacking xz-utils (5.2.5-2ubuntu1) ...
Selecting previously unselected package libdpkg-perl.
Preparing to unpack .../02-libdpkg-perl_1.21.1ubuntu2.2_all.deb ...
Unpacking libdpkg-perl (1.21.1ubuntu2.2) ...
Selecting previously unselected package lto-disabled-list.
Preparing to unpack .../03-lto-disabled-list_24_all.deb ...
Unpacking lto-disabled-list (24) ...
Selecting previously unselected package dpkg-dev.
Preparing to unpack .../04-dpkg-dev_1.21.1ubuntu2.2_all.deb ...
Unpacking dpkg-dev (1.21.1ubuntu2.2) ...
Selecting previously unselected package build-essential.
Preparing to unpack .../05-build-essential_12.9ubuntu3_amd64.deb ...
Unpacking build-essential (12.9ubuntu3) ...
Selecting previously unselected package libksba8:amd64.
Preparing to unpack .../06-libksba8_1.6.0-2ubuntu0.2_amd64.deb ...
Unpacking libksba8:amd64 (1.6.0-2ubuntu0.2) ...
Selecting previously unselected package dirmngr.
Preparing to unpack .../07-dirmngr_2.2.27-3ubuntu2.1_amd64.deb ...
Unpacking dirmngr (2.2.27-3ubuntu2.1) ...
Selecting previously unselected package libfakeroot:amd64.
Preparing to unpack .../08-libfakeroot_1.28-1ubuntu1_amd64.deb ...
Unpacking libfakeroot:amd64 (1.28-1ubuntu1) ...
Selecting previously unselected package fakeroot.
Preparing to unpack .../09-fakeroot_1.28-1ubuntu1_amd64.deb ...
Unpacking fakeroot (1.28-1ubuntu1) ...
Selecting previously unselected package gnupg-l10n.
Preparing to unpack .../10-gnupg-l10n_2.2.27-3ubuntu2.1_all.deb ...
Unpacking gnupg-l10n (2.2.27-3ubuntu2.1) ...
Selecting previously unselected package gnupg-utils.
Preparing to unpack .../11-gnupg-utils_2.2.27-3ubuntu2.1_amd64.deb ...
Unpacking gnupg-utils (2.2.27-3ubuntu2.1) ...
Selecting previously unselected package gpg-wks-client.
Preparing to unpack .../12-gpg-wks-client_2.2.27-3ubuntu2.1_amd64.deb ...
Unpacking gpg-wks-client (2.2.27-3ubuntu2.1) ...
Selecting previously unselected package gpg-wks-server.
Preparing to unpack .../13-gpg-wks-server_2.2.27-3ubuntu2.1_amd64.deb ...
Unpacking gpg-wks-server (2.2.27-3ubuntu2.1) ...
Selecting previously unselected package gpgsm.
Preparing to unpack .../14-gpgsm_2.2.27-3ubuntu2.1_amd64.deb ...
Unpacking gpgsm (2.2.27-3ubuntu2.1) ...
Selecting previously unselected package gnupg.
Preparing to unpack .../15-gnupg_2.2.27-3ubuntu2.1_all.deb ...
Unpacking gnupg (2.2.27-3ubuntu2.1) ...
Selecting previously unselected package libalgorithm-diff-perl.
Preparing to unpack .../16-libalgorithm-diff-perl_1.201-1_all.deb ...
Unpacking libalgorithm-diff-perl (1.201-1) ...
Selecting previously unselected package libalgorithm-diff-xs-perl.
Preparing to unpack .../17-libalgorithm-diff-xs-perl_0.04-6build3_amd64.deb ...
Unpacking libalgorithm-diff-xs-perl (0.04-6build3) ...
Selecting previously unselected package libalgorithm-merge-perl.
Preparing to unpack .../18-libalgorithm-merge-perl_0.08-3_all.deb ...
Unpacking libalgorithm-merge-perl (0.08-3) ...
Selecting previously unselected package libfile-fcntllock-perl.
Preparing to unpack .../19-libfile-fcntllock-perl_0.22-3build7_amd64.deb ...
Unpacking libfile-fcntllock-perl (0.22-3build7) ...
Setting up libksba8:amd64 (1.6.0-2ubuntu0.2) ...
Setting up lto-disabled-list (24) ...
Setting up libfile-fcntllock-perl (0.22-3build7) ...
Setting up libalgorithm-diff-perl (1.201-1) ...
Setting up gpgsm (2.2.27-3ubuntu2.1) ...
Setting up libfakeroot:amd64 (1.28-1ubuntu1) ...
Setting up fakeroot (1.28-1ubuntu1) ...
update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/fakeroot.1.gz because associated file /usr/share/man/man1/fakeroot-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/faked.1.gz because associated file /usr/share/man/man1/faked-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/es/man1/fakeroot.1.gz because associated file /usr/share/man/es/man1/fakeroot-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/es/man1/faked.1.gz because associated file /usr/share/man/es/man1/faked-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/fr/man1/fakeroot.1.gz because associated file /usr/share/man/fr/man1/fakeroot-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/fr/man1/faked.1.gz because associated file /usr/share/man/fr/man1/faked-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/sv/man1/fakeroot.1.gz because associated file /usr/share/man/sv/man1/fakeroot-sysv.1.gz (of link group fakeroot) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/sv/man1/faked.1.gz because associated file /usr/share/man/sv/man1/faked-sysv.1.gz (of link group fakeroot) doesn't exist
Setting up dirmngr (2.2.27-3ubuntu2.1) ...
Setting up gnupg-l10n (2.2.27-3ubuntu2.1) ...
Setting up xz-utils (5.2.5-2ubuntu1) ...
update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/lzma.1.gz because associated file /usr/share/man/man1/xz.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/unlzma.1.gz because associated file /usr/share/man/man1/unxz.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzcat.1.gz because associated file /usr/share/man/man1/xzcat.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzmore.1.gz because associated file /usr/share/man/man1/xzmore.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzless.1.gz because associated file /usr/share/man/man1/xzless.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzdiff.1.gz because associated file /usr/share/man/man1/xzdiff.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzcmp.1.gz because associated file /usr/share/man/man1/xzcmp.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzgrep.1.gz because associated file /usr/share/man/man1/xzgrep.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzegrep.1.gz because associated file /usr/share/man/man1/xzegrep.1.gz (of link group lzma) doesn't exist
update-alternatives: warning: skip creation of /usr/share/man/man1/lzfgrep.1.gz because associated file /usr/share/man/man1/xzfgrep.1.gz (of link group lzma) doesn't exist
Setting up libdpkg-perl (1.21.1ubuntu2.2) ...
Setting up gpg-wks-server (2.2.27-3ubuntu2.1) ...
Setting up libalgorithm-diff-xs-perl (0.04-6build3) ...
Setting up liblocale-gettext-perl (1.07-4build3) ...
Setting up libalgorithm-merge-perl (0.08-3) ...
Setting up gnupg-utils (2.2.27-3ubuntu2.1) ...
Setting up gpg-wks-client (2.2.27-3ubuntu2.1) ...
Setting up dpkg-dev (1.21.1ubuntu2.2) ...
Setting up gnupg (2.2.27-3ubuntu2.1) ...
Setting up build-essential (12.9ubuntu3) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...

linuxbrew@56816a3e97f7:~$ brew tests --only caveats
Randomized with seed 46876
1 process for 1 spec, ~ 1 spec per process
......................

Took 2 seconds
linuxbrew@56816a3e97f7:~$ brew config
HOMEBREW_VERSION: 4.0.28
ORIGIN: https://github.com/Homebrew/brew
HEAD: 2b4cfb033757b63ab0b128d01c58db76480956f9
Last commit: 11 days ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 9b18f7246118ad2669d7cc017978238472624af2
Core tap last commit: 10 days ago
Core tap branch: master
Core tap JSON: 17 Jul 01:48 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 1
Homebrew Ruby: 2.6.10 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby
CPU: single-core 64-bit skylake
Clang: N/A
Git: 2.41.0 => /bin/git
Curl: 7.81.0 => /bin/curl
Kernel: Linux 6.3.11-200.fc38.x86_64 x86_64 GNU/Linux
OS: Unknown
Host glibc: 2.35
/usr/bin/gcc: 11.3.0
/usr/bin/ruby: N/A
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A

However, in Linux, it still fails regardless. This might have to do with Podman's VM (on macOS). Because in a docker --rm -it homebrew/brew (on macOS), installing build-essential does not fix it:

linuxbrew@35f1d2e2e853:~$ sudo apt-get update
Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:2 https://cli.github.com/packages stable InRelease                         
Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease                         
Hit:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease                 
Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease               
Hit:6 https://ppa.launchpadcontent.net/git-core/ppa/ubuntu jammy InRelease     
Reading package lists... Done
linuxbrew@35f1d2e2e853:~$ sudo apt-get install build-essential procps curl file git
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
build-essential is already the newest version (12.9ubuntu3).
file is already the newest version (1:5.41-3).
procps is already the newest version (2:3.3.17-6ubuntu2).
curl is already the newest version (7.81.0-1ubuntu1.10).
git is already the newest version (1:2.41.0-0ppa1~ubuntu22.04.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
linuxbrew@35f1d2e2e853:~$ brew tests --only caveats
Randomized with seed 45124
1 process for 1 spec, ~ 1 spec per process
.........F............

Failures:

  1) Caveats#caveats when f.keg_only is not nil when joining different caveat types together adds the correct amount of new lines to the output
     Failure/Error: expect(caveats).to include("if you don't want/need a background service")

       expected "something else\n\nformula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230717-4879-tzorkt/prefix,\nbecause some reason.\n\nWarning: formula_name provides a service which can only be used on macOS or systemd!\nYou can manually execute the service instead with:\n  /tmp/homebrew-tests-20230717-4879-tzorkt/cellar/formula_name/1.0/bin/cmd\n" to include "if you don't want/need a background service"
       Diff:
       @@ -1,8 +1,15 @@
       -if you don't want/need a background service
       +something else
       +
       +formula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230717-4879-tzorkt/prefix,
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230717-4879-tzorkt/cellar/formula_name/1.0/bin/cmd

     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'

Took 3 seconds
Tests Failed
linuxbrew@35f1d2e2e853:~$ brew doctor
Your system is ready to brew.
linuxbrew@35f1d2e2e853:~$ brew config
HOMEBREW_VERSION: 4.0.28-72-gbd5bd84
ORIGIN: https://github.com/Homebrew/brew
HEAD: bd5bd8437f96bce1576fbc46d64203f41fd848c7
Last commit: 21 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 9b18f7246118ad2669d7cc017978238472624af2
Core tap last commit: 10 days ago
Core tap branch: master
Core tap JSON: 17 Jul 02:11 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 2
Homebrew Ruby: 2.6.10 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby
CPU: dual-core 64-bit skylake
Clang: N/A
Git: 2.41.0 => /bin/git
Curl: 7.81.0 => /bin/curl
Kernel: Linux 5.15.49-linuxkit-pr x86_64 GNU/Linux
OS: Unknown
Host glibc: 2.35
/usr/bin/gcc: 11.3.0
/usr/bin/ruby: N/A
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A

Not sure what this means. Any ideas why it might fail depending on the host, even while using the same container?

osalbahr commented 1 year ago

I tried in a fedora container, but I get a different error:

[linuxbrew@cc2ad233dc93 /]$ brew tests --only caveats
Full log

[linuxbrew@cc2ad233dc93 /]$ brew tests --only caveats
Fetching gem metadata from https://rubygems.org/.......
Using concurrent-ruby 1.2.2
Using minitest 5.18.1
Using zeitwerk 2.6.8
Using public_suffix 5.0.1
Using ast 2.4.2
Using bindata 2.4.15
Using msgpack 1.7.1
Using bundler 2.3.26
Using byebug 11.1.3
Using coderay 1.1.3
Using did_you_mean 1.6.3
Using diff-lcs 1.5.0
Using hana 1.3.7
Using hpricot 0.8.6
Using regexp_parser 2.8.1
Using docile 1.4.0
Using mustache 1.1.1
Using parallel 1.23.0
Using racc 1.7.1
Using plist 3.7.0
Using rack 3.0.8
Using rainbow 3.1.1
Using rdiscount 2.2.7.1
Using rexml 3.2.5
Using rspec-support 3.12.1
Using sorbet-runtime 0.5.10461
Using ruby-progressbar 1.13.0
Using unicode-display_width 2.4.2
Using ruby-macho 3.0.0
Using ruby-prof 1.4.3
Using simplecov-html 0.12.3
Using simplecov_json_formatter 0.1.4
Using json 2.6.3
Using warning 1.3.0
Using i18n 1.14.1
Using tzinfo 2.0.6
Using addressable 2.8.4
Using bootsnap 1.16.0
Using elftools 1.2.0
Using parallel_tests 3.13.0
Using parser 3.2.2.3
Using ronn 0.7.3
Using rspec-core 3.12.2
Using rspec-expectations 3.12.3
Using rspec-mocks 3.12.5
Using rspec-sorbet 1.9.2
Using method_source 1.0.0
Using stackprof 0.2.25
Using activesupport 6.1.7.4
Using rspec 3.12.0
Using rspec-github 2.4.0
Using patchelf 1.4.0
Using rspec-retry 0.6.2
Using rspec_junit_formatter 0.6.0
Using rubocop-ast 1.29.0
Using simplecov 0.22.0
Using pry 0.14.2
Using rspec-its 1.3.0
Using rubocop 1.50.2
Using simplecov-cobertura 2.1.0
Using rubocop-capybara 2.18.0
Using rubocop-performance 1.17.1
Using rubocop-rails 2.19.1
Using rubocop-sorbet 0.7.0
Using rubocop-rspec 2.20.0
Installing unf_ext 0.0.8.2 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/unf_ext-0.0.8.2/ext/unf_ext
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby -I
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0 -r ./siteconf20230717-4500-22zru2.rb extconf.rb
checking for -lstdc++... no
creating Makefile

current directory: /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/unf_ext-0.0.8.2/ext/unf_ext
make "DESTDIR=" clean

current directory: /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/unf_ext-0.0.8.2/ext/unf_ext
make "DESTDIR="
compiling unf.cc
make: c++: No such file or directory
make: *** [Makefile:213: unf.o] Error 127

make failed, exit code 2

Gem files will remain installed in /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/unf_ext-0.0.8.2 for inspection.
Results logged to
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/extensions/x86_64-linux/2.6.0-static/unf_ext-0.0.8.2/gem_make.out

  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/builder.rb:99:in `run'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/builder.rb:51:in `block in make'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/builder.rb:43:in `each'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/builder.rb:43:in `make'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/ext_conf_builder.rb:62:in `block in build'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/tempfile.rb:295:in `open'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/ext_conf_builder.rb:29:in `build'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/builder.rb:185:in `block in build_extension'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/builder.rb:181:in `build_extension'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/builder.rb:229:in `block in build_extensions'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/builder.rb:226:in `each'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/ext/builder.rb:226:in `build_extensions'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/lib/ruby/2.6.0/rubygems/installer.rb:830:in `build_extensions'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/rubygems_gem_installer.rb:72:in
`build_extensions'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/rubygems_gem_installer.rb:28:in `install'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/source/rubygems.rb:207:in `install'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/installer/gem_installer.rb:54:in `install'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/installer/parallel_installer.rb:186:in
`do_install'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/installer/parallel_installer.rb:177:in `block
in worker_pool'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/worker.rb:62:in `apply_func'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/worker.rb:57:in `block in process_queue'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/worker.rb:54:in `loop'
  /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/worker.rb:54:in `process_queue'
/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/gems/bundler-2.3.26/lib/bundler/worker.rb:91:in `block (2 levels) in
create_threads'

Error summary:

An error occurred while installing unf_ext (0.0.8.2), and Bundler cannot continue.

In Gemfile:
  json_schemer was resolved to 1.0.3, which depends on
    simpleidn was resolved to 0.2.1, which depends on
      unf was resolved to 0.1.4, which depends on
        unf_ext
Error: failed to run `/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/bin/bundle install`!
[linuxbrew@cc2ad233dc93 /]$ /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/bin/bundle install
Could not locate Gemfile
[linuxbrew@cc2ad233dc93 /]$ brew doctor
Your system is ready to brew.
[linuxbrew@cc2ad233dc93 /]$ brew config
HOMEBREW_VERSION: 4.0.28
ORIGIN: https://github.com/Homebrew/brew
HEAD: 2b4cfb033757b63ab0b128d01c58db76480956f9
Last commit: 11 days ago
Core tap JSON: 17 Jul 02:43 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 1
Homebrew Ruby: 2.6.10 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby
CPU: single-core 64-bit skylake
Clang: N/A
Git: 2.41.0 => /bin/git
Curl: 8.0.1 => /bin/curl
Kernel: Linux 6.3.11-200.fc38.x86_64 x86_64 GNU/Linux
OS: Fedora release 38 (Thirty Eight)
Host glibc: 2.37
/usr/bin/gcc: 13.1.1
/usr/bin/ruby: 3.2.2
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A

Not sure what this means.

osalbahr commented 1 year ago

The same ruby error happens in ubuntu:

linuxbrew@7389d78c4277:/$ brew install-bundler-gems 2>&1 | tail

An error occurred while installing unf_ext (0.0.8.2), and Bundler cannot
continue.

In Gemfile:
  json_schemer was resolved to 1.0.3, which depends on
    simpleidn was resolved to 0.2.1, which depends on
      unf was resolved to 0.1.4, which depends on
        unf_ext
Error: failed to run `/home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/bin/bundle install`!
linuxbrew@7389d78c4277:/$ /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/bundle/ruby/2.6.0/bin/bundle install
Could not locate Gemfile
linuxbrew@7389d78c4277:/$ brew doctor
Your system is ready to brew.
linuxbrew@7389d78c4277:/$ brew config
HOMEBREW_VERSION: 4.0.28
ORIGIN: https://github.com/Homebrew/brew
HEAD: 2b4cfb033757b63ab0b128d01c58db76480956f9
Last commit: 11 days ago
Core tap JSON: 17 Jul 03:18 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 1
Homebrew Ruby: 2.6.10 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby
CPU: single-core 64-bit skylake
Clang: N/A
Git: 2.34.1 => /bin/git
Curl: 7.81.0 => /bin/curl
Kernel: Linux 6.3.11-200.fc38.x86_64 x86_64 GNU/Linux
OS: Unknown
Host glibc: 2.35
/usr/bin/gcc: 11.3.0
/usr/bin/ruby: 3.0.2
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A
linuxbrew@7389d78c4277:/$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Not sure which step in Homebrew's Dockerfile handles this.

MikeMcQuaid commented 1 year ago

Happy to keep helping in this issue but closing because it's working in our CI.

osalbahr commented 1 year ago

@MikeMcQuaid Are you able to replicate this error locally? So far the test seems to only reliably pass on macOS.

Also, since the issue is closed now, would you suggest I open a discussion at https://github.com/orgs/Homebrew/discussions ?

MikeMcQuaid commented 1 year ago

@osalbahr I cannot replicate locally and this test is passing in multiple configurations in the Linux CI on this repository.

Bo98 commented 1 year ago

In the Linux case, I believe it happens if systemctl is not available. The test should ideally mock pretending that it is available.

osalbahr commented 1 year ago

I think it has to do with the way brew config works perhaps? (as in #15689). Does brew tests use a similar mechanism to brew config to determine the OS type? I think the tests --only caveats passing depends on whether tests knows that I'm not on a macOS.

$ podman run --rm -it homebrew/brew
linuxbrew@2e7eff35d8e0:~$ brew config | grep "OS:"
OS: Unknown
linuxbrew@2e7eff35d8e0:~$ brew tests --only caveats &> >(tail)
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-762-1270euf/cellar/formula_name/1.0/bin/cmd
     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'
Tests Failed

Took 3 seconds
linuxbrew@2e7eff35d8e0:~$ (sudo apt-get update && sudo apt-get install -y lsb-release) &> >(tail)
Setting up distro-info-data (0.52ubuntu0.4) ...
Setting up libmpdec3:amd64 (2.5.1-2build2) ...
Setting up libpython3.10-stdlib:amd64 (3.10.6-1~22.04.2ubuntu1.1) ...
Setting up libpython3-stdlib:amd64 (3.10.6-1~22.04) ...
Setting up python3.10 (3.10.6-1~22.04.2ubuntu1.1) ...
Setting up python3 (3.10.6-1~22.04) ...
running python rtupdate hooks for python3.10...
running python post-rtupdate hooks for python3.10...
Setting up lsb-release (11.1.0ubuntu4) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
linuxbrew@2e7eff35d8e0:~$ brew config | grep "OS:"
OS: Ubuntu 22.04.2 LTS (jammy)
linuxbrew@2e7eff35d8e0:~$ brew tests --only caveats &> >(tail)
linuxbrew@2e7eff35d8e0:~$ Randomized with seed 6129
1 process for 1 spec, ~ 1 spec per process
......................

Took 2 seconds
osalbahr commented 1 year ago

Ok, I think I've confirmed my guess:

$ podman run --rm -it homebrew/brew
linuxbrew@db4aa289008b:~$ brew tests --only caveats &> >(tail) || echo $?
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-378-iffhsp/cellar/formula_name/1.0/bin/cmd
     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'
Tests Failed

Took 3 seconds
1
linuxbrew@db4aa289008b:~$ (sudo apt-get update && sudo apt-get install -y lsb-release) &> >(tail) && echo $?
Setting up distro-info-data (0.52ubuntu0.4) ...
Setting up libmpdec3:amd64 (2.5.1-2build2) ...
Setting up libpython3.10-stdlib:amd64 (3.10.6-1~22.04.2ubuntu1.1) ...
Setting up libpython3-stdlib:amd64 (3.10.6-1~22.04) ...
Setting up python3.10 (3.10.6-1~22.04.2ubuntu1.1) ...
Setting up python3 (3.10.6-1~22.04) ...
running python rtupdate hooks for python3.10...
running python post-rtupdate hooks for python3.10...
Setting up lsb-release (11.1.0ubuntu4) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
0
linuxbrew@db4aa289008b:~$ brew tests --only caveats && echo $?           
Randomized with seed 4349
1 process for 1 spec, ~ 1 spec per process
......................

Took 3 seconds
0

I see two ways going about this:

  1. Include lsb-release in homebrew/brew (20.8 MB of additional disk space, and consequently a few more seconds in CI). This is probably not worth it.
  2. Wait until linux.rb falls back to $HOMEBREW_OS_VERSION ($PRETTY_NAME) before going to Unknown and see if that fixes it. I think it probably will, I will see.

My goal is that brew tests should pass off the bat when the requirements in "Homebrew on Linux" are satisfied (in a best-effort manner). Earlier I'd just ignore brew tests if I don't think the error is relevant, which is not ideal.

osalbahr commented 1 year ago

In the Linux case, I believe it happens if systemctl is not available. The test should ideally mock pretending that it is available.

I'm not sure if this is the case, at least not in podman run -it homebrew/brew on my machine. I also accidentally discovered that redirecting the output of brew tests --only caveats to tee, or tail, breaks it, but not cat? I have no idea why. Anyways, lsb_release is needed regardless of the redirection.

Logs (with lsb_release)
$ podman run --rm -it homebrew/brew
linuxbrew@a63dd38daaae:~$ command -v systemctl || echo $?                
1
linuxbrew@a63dd38daaae:~$ command -v lsb_release || echo $?
1
linuxbrew@a63dd38daaae:~$ brew config | grep 'OS'
OS: Unknown
linuxbrew@a63dd38daaae:~$ brew tests --only caveats &> >(tail) || echo $?
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-762-1165pi2/cellar/formula_name/1.0/bin/cmd
     # ./test/caveats_spec.rb:289:in `block (5 levels) in '
Tests Failed

Took 3 seconds
1
linuxbrew@a63dd38daaae:~$ (sudo apt-get update && sudo apt-get install -y lsb-release) &> >(tail) && echo $?
Setting up distro-info-data (0.52ubuntu0.4) ...
Setting up libmpdec3:amd64 (2.5.1-2build2) ...
Setting up libpython3.10-stdlib:amd64 (3.10.6-1~22.04.2ubuntu1.1) ...
Setting up libpython3-stdlib:amd64 (3.10.6-1~22.04) ...
Setting up python3.10 (3.10.6-1~22.04.2ubuntu1.1) ...
Setting up python3 (3.10.6-1~22.04) ...
running python rtupdate hooks for python3.10...
running python post-rtupdate hooks for python3.10...
Setting up lsb-release (11.1.0ubuntu4) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
0
linuxbrew@a63dd38daaae:~$ command -v systemctl || echo $?
1
linuxbrew@a63dd38daaae:~$ command -v lsb_release && echo $?
/usr/bin/lsb_release
0
linuxbrew@a63dd38daaae:~$ brew config | grep 'OS'
OS: Ubuntu 22.04.2 LTS (jammy)
linuxbrew@a63dd38daaae:~$ brew tests --only caveats &> >(tail) && echo $?
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
linuxbrew@a63dd38daaae:~$        +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-1869-3d5i0u/cellar/formula_name/1.0/bin/cmd
     # ./test/caveats_spec.rb:289:in `block (5 levels) in '
Tests Failed

Took 3 seconds

linuxbrew@a63dd38daaae:~$ brew tests --only caveats && echo $?           
Randomized with seed 34426
1 process for 1 spec, ~ 1 spec per process
......................

Took 2 seconds
0
linuxbrew@a63dd38daaae:~$ brew tests --only caveats > /dev/stdout
Randomized with seed 25298
1 process for 1 spec, ~ 1 spec per process
......................

Took 1 seconds
linuxbrew@a63dd38daaae:~$ brew tests --only caveats | tee        
Randomized with seed 63853
1 process for 1 spec, ~ 1 spec per process
......F...............

Failures:

  1) Caveats#caveats when f.keg_only is not nil when joining different caveat types together adds the correct amount of new lines to the output
     Failure/Error: expect(caveats).to include("if you don't want/need a background service")

       expected "something else\n\nformula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230718-2538-1n2zp5q/prefix,\nbecause some reason.\n\nWarning: formula_name provides a service which can only be used on macOS or systemd!\nYou can manually execute the service instead with:\n  /tmp/homebrew-tests-20230718-2538-1n2zp5q/cellar/formula_name/1.0/bin/cmd\n" to include "if you don't want/need a background service"
       Diff:
       @@ -1,8 +1,15 @@
       -if you don't want/need a background service
       +something else
       +
       +formula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230718-2538-1n2zp5q/prefix,
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-2538-1n2zp5q/cellar/formula_name/1.0/bin/cmd
     # ./test/caveats_spec.rb:289:in `block (5 levels) in '
Tests Failed

Took 3 seconds
linuxbrew@a63dd38daaae:~$ brew tests --only caveats | cat
Randomized with seed 20552
1 process for 1 spec, ~ 1 spec per process
......................

Took 2 seconds
linuxbrew@a63dd38daaae:~$ brew tests --only caveats | tail
Tests Failed
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-2985-16a9ez4/cellar/formula_name/1.0/bin/cmd
     # ./test/caveats_spec.rb:289:in `block (5 levels) in '

Took 3 seconds

Or.... maybe the redirection is irrelevant. Heisenbug?

linuxbrew@a63dd38daaae:~$ brew tests --only caveats | tail
Randomized with seed 17213
1 process for 1 spec, ~ 1 spec per process
......................

Took 1 seconds
linuxbrew@a63dd38daaae:~$ brew tests --only caveats | tail
Tests Failed
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-3800-oc7fmr/cellar/formula_name/1.0/bin/cmd
     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'

Took 3 seconds
linuxbrew@a63dd38daaae:~$ brew tests --only caveats | tail
Randomized with seed 8588
1 process for 1 spec, ~ 1 spec per process
......................

Took 2 seconds
linuxbrew@a63dd38daaae:~$ brew tests --only caveats | tail
Tests Failed
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-4247-v7btp9/cellar/formula_name/1.0/bin/cmd
     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'

Took 3 seconds
linuxbrew@a63dd38daaae:~$ brew tests --only caveats | tail
Tests Failed
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-4470-1imsgf3/cellar/formula_name/1.0/bin/cmd
     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'

Took 3 seconds

I thought containers were meant to be reproducible, and they are for the most part, but I guess not perfectly. It seems to depend on the host, or maybe whether I use podman or docker (I'd have to cross-check my dual-boot of macOS and Fedora Silverblue. The tests above were using podman on Silverblue):

$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="38.20230717.0 (Silverblue)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora Linux 38.20230717.0 (Silverblue)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://silverblue.fedoraproject.org"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-silverblue/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://github.com/fedora-silverblue/issue-tracker/issues"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="Silverblue"
VARIANT_ID=silverblue
OSTREE_VERSION='38.20230717.0'
osalbahr commented 1 year ago

Ok, it seems to sometimes pass even without lsb_release, but maybe less often. I'm confused.

linuxbrew@b618bac11461:~$ brew tests --only caveats
Randomized with seed 54923
1 process for 1 spec, ~ 1 spec per process
......................

Took 1 seconds
linuxbrew@b618bac11461:~$ brew tests --only caveats
Randomized with seed 27217
1 process for 1 spec, ~ 1 spec per process
..................F...

Failures:

  1) Caveats#caveats when f.keg_only is not nil when joining different caveat types together adds the correct amount of new lines to the output
     Failure/Error: expect(caveats).to include("if you don't want/need a background service")

       expected "something else\n\nformula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230718-1265-1m6eibu/prefix,\nbecause some reason.\n\nWarning: formula_name provides a service which can only be used on macOS or systemd!\nYou can manually execute the service instead with:\n  /tmp/homebrew-tests-20230718-1265-1m6eibu/cellar/formula_name/1.0/bin/cmd\n" to include "if you don't want/need a background service"
       Diff:
       @@ -1,8 +1,15 @@
       -if you don't want/need a background service
       +something else
       +
       +formula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230718-1265-1m6eibu/prefix,
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-1265-1m6eibu/cellar/formula_name/1.0/bin/cmd

     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'

Took 3 seconds
Tests Failed
linuxbrew@b618bac11461:~$ lsb_release
bash: lsb_release: command not found
osalbahr commented 1 year ago

Ok, it's the seed. But why?

linuxbrew@b618bac11461:~$ brew tests --only caveats --seed 54923 
Randomized with seed 54923
1 process for 1 spec, ~ 1 spec per process
......................

Took 1 seconds
linuxbrew@b618bac11461:~$ brew tests --only caveats --seed 54923
Randomized with seed 54923
1 process for 1 spec, ~ 1 spec per process
......................

Took 2 seconds
linuxbrew@b618bac11461:~$ brew tests --only caveats --seed 54923
Randomized with seed 54923
1 process for 1 spec, ~ 1 spec per process
......................

Took 1 seconds
linuxbrew@b618bac11461:~$ brew tests --only caveats --seed 54924
Randomized with seed 54924
1 process for 1 spec, ~ 1 spec per process
....F.................

Failures:

  1) Caveats#caveats when f.keg_only is not nil when joining different caveat types together adds the correct amount of new lines to the output
     Failure/Error: expect(caveats).to include("if you don't want/need a background service")

       expected "something else\n\nformula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230718-4534-1jgcimp/prefix,\nbecause some reason.\n\nWarning: formula_name provides a service which can only be used on macOS or systemd!\nYou can manually execute the service instead with:\n  /tmp/homebrew-tests-20230718-4534-1jgcimp/cellar/formula_name/1.0/bin/cmd\n" to include "if you don't want/need a background service"
       Diff:
       @@ -1,8 +1,15 @@
       -if you don't want/need a background service
       +something else
       +
       +formula_name is keg-only, which means it was not symlinked into /tmp/homebrew-tests-20230718-4534-1jgcimp/prefix,
       +because some reason.
       +
       +Warning: formula_name provides a service which can only be used on macOS or systemd!
       +You can manually execute the service instead with:
       +  /tmp/homebrew-tests-20230718-4534-1jgcimp/cellar/formula_name/1.0/bin/cmd

     # ./test/caveats_spec.rb:289:in `block (5 levels) in <top (required)>'

Took 3 seconds
Tests Failed
osalbahr commented 1 year ago

I am now pretty confident that the seed has to do with the error. Could you check if you get a similar output on your end using seeder.sh either directly on your machine or in a container?

linuxbrew@9d10b23b965d:~$ brew doctor
Your system is ready to brew.
linuxbrew@9d10b23b965d:~$ time ./seeder.sh 
Tue Jul 18 18:55:26 UTC 2023
seed = 54923: 0 failures out of 10
seed = 54924: 10 failures out of 10
seed = null: 8 failures out of 10

real    2m10.897s
user    1m21.188s
sys 0m30.464s
linuxbrew@9d10b23b965d:~$ brew config
HOMEBREW_VERSION: 4.0.29
ORIGIN: https://github.com/Homebrew/brew
HEAD: a2a05f96ef6215643f599e7bfeaacd4f34e7cd9a
Last commit: 10 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: efc626a3fe39ef940c5db0c7358ab31c0adb92ab
Core tap last commit: 11 hours ago
Core tap branch: master
Core tap JSON: 18 Jul 18:55 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.10 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby
CPU: quad-core 64-bit skylake
Clang: N/A
Git: 2.41.0 => /bin/git
Curl: 7.81.0 => /bin/curl
Kernel: Linux 6.3.12-200.fc38.x86_64 x86_64 GNU/Linux
OS: Unknown
Host glibc: 2.35
/usr/bin/gcc: 11.3.0
/usr/bin/ruby: N/A
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A

The script: seeder.sh.txt

#!/usr/bin/env bash

# File: seeder.sh
#
# Test `brew tests` with seeds:
#   54923
#   54924
#   (and seedless)
#
# See https://github.com/Homebrew/brew/issues/15644

date | tee -a seeder.log

limit=10

# Comment out for all `tests` (warning: takes an eternity)
test='--only caveats'

# Seeds
for seed in {54923,54924}
do
    fail=0

    for ((i=0;i<limit;i++))
    do
        brew tests --seed "$seed" $test &>>seeder.log || fail=$((fail+1))
    done

    echo "seed = $seed: $fail failures out of $limit" | tee -a seeder.log
done

# No seed
fail=0

for ((i=0;i<limit;i++))
do
    brew tests &>>seeder.log $test || fail=$((fail+1))
done

echo "seed = null: $fail failures out of 10" | tee -a seeder.log

Log file: seeder-full.log


Edit: it happens in master too.

Logs (master)
linuxbrew@919d0239fde7:~$ brew doctor
Your system is ready to brew.
linuxbrew@919d0239fde7:~$ brew config
HOMEBREW_VERSION: 4.0.28-126-g93ba812
ORIGIN: https://github.com/Homebrew/brew
HEAD: 93ba81208c608cd5110499784e104d2acbb6ee5c
Last commit: 3 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: efc626a3fe39ef940c5db0c7358ab31c0adb92ab
Core tap last commit: 12 hours ago
Core tap branch: master
Core tap JSON: 18 Jul 19:29 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.10 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby
CPU: quad-core 64-bit skylake
Clang: N/A
Git: 2.41.0 => /bin/git
Curl: 7.81.0 => /bin/curl
Kernel: Linux 6.3.12-200.fc38.x86_64 x86_64 GNU/Linux
OS: Unknown
Host glibc: 2.35
/usr/bin/gcc: 11.3.0
/usr/bin/ruby: N/A
glibc: N/A
gcc@11: N/A
gcc: N/A
xorg: N/A
linuxbrew@919d0239fde7:~$ time ./seeder.sh 
Tue Jul 18 19:30:17 UTC 2023
seed = 54923: 0 failures out of 10
seed = 54924: 10 failures out of 10
seed = null: 7 failures out of 10

real    2m13.918s
user    1m23.772s
sys 0m31.515s
linuxbrew@919d0239fde7:~$ curl https://raw.githubusercontent.com/hykilpikonna/hyfetch/master/neofetch | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  495k  100  495k    0     0   579k      0 --:--:-- --:--:-- --:--:--  578k
                             ....             linuxbrew@919d0239fde7 
              .',:clooo:  .:looooo:.          ---------------------- 
           .;looooooooc  .oooooooooo'         OS: Ubuntu 22.04.2 LTS x86_64 
        .;looooool:,''.  :ooooooooooc         Host: MacBookPro14,1 1.0 
       ;looool;.         'oooooooooo,         Kernel: 6.3.12-200.fc38.x86_64 
      ;clool'             .cooooooc.  ,,      Uptime: 2 hours, 30 mins 
         ...                ......  .:oo,     Packages: 200 (dpkg) 
  .;clol:,.                        .loooo'    Shell: bash 5.1.16 
 :ooooooooo,                        'ooool    Resolution: 2560x1600 
'ooooooooooo.                        loooo.   Terminal: gtk-cursor-theme-name 
'ooooooooool                         coooo.   CPU: Intel i5-7360U (4) @ 3.6GHz 
 ,loooooooc.                        .loooo.   Memory: 3.89 GiB / 7.61 GiB (51%) 
   .,;;;'.                          ;ooooc    BIOS: Apple Inc. 0.1 (04/05/2023) 
       ...                         ,ooool.
    .cooooc.              ..',,'.  .cooo.                             
      ;ooooo:.           ;oooooooc.  :l.                              
       .coooooc,..      coooooooooo.
         .:ooooooolc:. .ooooooooooo'
           .':loooooo;  ,oooooooooc
               ..';::c'  .;loooo:'
                             .

linuxbrew@919d0239fde7:~$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
linuxbrew@919d0239fde7:~$ history
    1  sudo apt-get update
    2  sudo apt-get upgrade
    3  cd $(brew --repo)
    4  git checkout master
    5  git pull
    6  brew update
    7  cd
    8  cat > seeder.sh <<\EOF
#!/usr/bin/env bash

# File: seeder.sh
#
# Test `brew tests` with seeds:
#   54923
#   54924
#   (and seedless)
#
# See https://github.com/Homebrew/brew/issues/15644

date | tee -a seeder.log

limit=10

# Comment out for all `tests` (warning: takes an eternity)
test='--only caveats'

# Seeds
for seed in {54923,54924}
do
    fail=0

    for ((i=0;i>seeder.log || fail=$((fail+1))
    done

    echo "seed = $seed: $fail failures out of $limit" | tee -a seeder.log
done

# No seed
fail=0

for ((i=0;i>seeder.log $test || fail=$((fail+1))
done

echo "seed = null: $fail failures out of 10" | tee -a seeder.log

EOF

    9  chmod +x seeder.sh 
   10  brew doctor
   11  brew developer on
   12  brew doctor
   13  brew config
   14  time ./seeder.sh 
   15  curl https://raw.githubusercontent.com/hykilpikonna/hyfetch/master/neofetch | bash
   16  cat /etc/os-release 
   17  history
osalbahr commented 1 year ago

In the Linux case, I believe it happens if systemctl is not available. The test should ideally mock pretending that it is available.

Sorry, yes, that does it. But only if I apt-get install it.

With apt-get install systemd:

linuxbrew@e6c607476bca:~$ ./seeder.sh 
Wed Jul 19 00:47:11 UTC 2023
seed = 54923: 0 failures out of 10
seed = 54924: 10 failures out of 10
seed = null: 9 failures out of 10
linuxbrew@e6c607476bca:~$ (sudo apt-get update && sudo apt-get install -y systemd) &> >(tail)
Created symlink /etc/systemd/system/sysinit.target.wants/systemd-pstore.service → /lib/systemd/system/systemd-pstore.service.
Setting up python3-gi (3.42.1-0ubuntu1) ...
Setting up shared-mime-info (2.1-2) ...
Setting up systemd-timesyncd (249.11-0ubuntu3.9) ...
Created symlink /etc/systemd/system/dbus-org.freedesktop.timesync1.service → /lib/systemd/system/systemd-timesyncd.service.
Created symlink /etc/systemd/system/sysinit.target.wants/systemd-timesyncd.service → /lib/systemd/system/systemd-timesyncd.service.
Setting up python3-dbus (1.2.18-3build1) ...
Setting up networkd-dispatcher (2.1-2ubuntu0.22.04.2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/networkd-dispatcher.service → /lib/systemd/system/networkd-dispatcher.service.
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
linuxbrew@e6c607476bca:~$ ./seeder.sh 
Wed Jul 19 00:51:48 UTC 2023
seed = 54923: 0 failures out of 10
seed = 54924: 0 failures out of 10
seed = null: 0 failures out of 10

With brew install systemd:

linuxbrew@82202354933c:~$ ./seeder.sh 
Wed Jul 19 00:59:11 UTC 2023
seed = 54923: 0 failures out of 10
seed = 54924: 10 failures out of 10
seed = null: 7 failures out of 10
linuxbrew@82202354933c:~$ which systemctl                
linuxbrew@82202354933c:~$ brew install systemd &> >(tail)
  /home/linuxbrew/.linuxbrew/etc/bash_completion.d
==> Summary
🍺  /home/linuxbrew/.linuxbrew/Cellar/systemd/253.7: 663 files, 22.4MB
==> Running `brew cleanup systemd`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Caveats
==> systemd
Bash completion has been installed to:
  /home/linuxbrew/.linuxbrew/etc/bash_completion.d
linuxbrew@82202354933c:~$ which systemctl
/home/linuxbrew/.linuxbrew/bin/systemctl
linuxbrew@82202354933c:~$ ./seeder.sh 
Wed Jul 19 01:04:43 UTC 2023
seed = 54923: 0 failures out of 10
seed = 54924: 10 failures out of 10
seed = null: 8 failures out of 10

A couple questions remain. Let me know if any are outside the scope of Homebrew:

  1. Why did the seed matter?
  2. Why didn't brew install systemd suffice?

Also, what should I use to test changes to brew's functionality? Should I use homebrew/brew after manually apt installing systemd or is there another image better suited for development where brew tests passes off the bat? I assumed homebrew/brew was the one used in Homebrew's CI.


Side note: I apologize for the frequent comments. I should share the findings less often to avoid spamming emails.

Bo98 commented 1 year ago
  1. Why did the seed matter?

It shouldn't, so is probably a bug somewhere. A common reason is caches not being cleared properly, meaning the order of operations could affect the result of later tests. Parallelisation is another cause sometimes.

  1. Why didn't brew install systemd suffice?

It probably only searches the filtered PATH, which doesn't include HOMEBREW_PREFIX/bin.

Should I use homebrew/brew after manually apt installing systemd

homebrew/brew should be fine. The tests not working without systemd is a bug really, even if the caveat output is correct for installs without it. You may need systemd installed if you ever do any work with the Homebrew/services tap however.

brew CI doesn't use this image because it is built and pushed after merge, and would be too slow to rebuild every time before CI runs. The gains from doing so would be minimal given GitHub Actions ship with Linux images that have Homebrew installed. homebrew-core CI however uses the Docker imges (or well, homebrew/ubuntu22.04:master to be precise).

With the Docker images, it is however worth noting how the latest tag points to the latest release, while master points to the latest git commit on the master branch. Only the homebrew/ubuntu22.04 image offers the master tag.


Anyhow in terms of the bug itself, a quick glance shows that the context of the failing test (titled "when joining different caveat types together") is missing a:

      before do
        allow(Utils::Service).to receive(:launchctl?).and_return(true)
        allow(Utils::Service).to receive(:systemctl?).and_return(true)
      end
osalbahr commented 1 year ago

Thx! It's definitely useful to know about homebrew/ubuntu22.04:master so that I don't have to manually do the cd $(brew --repo) && git checkout master && git pull spiel (I've had issues with git checkout master && brew update on homebrew/brew). I should perhaps note that the same heisenbug happens in homebrew/ubuntu22.04:master.

GitHub Actions ship with Linux images that have Homebrew installed

Assuming you're referring to the GitHub-hosted runners, do you know an easy way to podman run ... such images? I know grep does show images that match the names in their website.

$ grep -r "runs-on: [a-z0-9-]" "$(brew --repo)/.github" | awk -F ' ' '{print $3}' | sort -u
macos-13
ubuntu-20.04
ubuntu-22.04
ubuntu-latest

Ubuntu2204-Readme.md includes "Systemd version: 249.11-0ubuntu3.9" but I can't find the actual image. The closest I can find is a Stackoverflow post that basically says "build it yourself or use a community built image" which is not really ideal to mimic Homebrew’s CI.


I'm not that well-versed in Ruby tbh to know where that snippet should go (or understand in detail what it really does). It seems that you found the bug and a possible fix. Let me know if I can help test or refine the patch.