search

Homebrew / brew

🍺 The missing package manager for macOS (or Linux)
https://brew.sh
BSD 2-Clause "Simplified" License
41.51k stars 9.76k forks source link

on_system conditionals not respected for `:all` manifest tab cause issues in keg relocation #18695

Open calvinit opened 3 weeks ago

calvinit commented 3 weeks ago

brew gist-logs <formula> link OR brew config AND brew doctor output

# brew config

HOMEBREW_VERSION: 4.4.3-60-g745596b
ORIGIN: https://github.com/Homebrew/brew
HEAD: 745596b55579f3b9acb198f6aaa4c8aba0ef9844
Last commit: 5 hours ago
Core tap HEAD: bff87d7ce5eb665fdd9b28e1ff8ee510d8cf352b
Core tap last commit: 2 hours ago
Core tap branch: openjdk@8_1.8.0-432
Core tap JSON: 01 Nov 06:23 UTC
Core cask tap origin: https://github.com/Homebrew/homebrew-cask.git
Core cask tap HEAD: 649f802bb9d4ba0a703685b982d661b570173bee
Core cask tap last commit: 6 hours ago
Core cask tap JSON: 01 Nov 06:23 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_BOTTLE_DOMAIN: https://mirrors.tuna.tsinghua.edu.cn/homebrew-bottles
HOMEBREW_CASK_OPTS: []
HOMEBREW_EDITOR: code
HOMEBREW_MAKE_JOBS: 12
HOMEBREW_NO_ENV_HINTS: set
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 3.3.5 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.3.5/bin/ruby
CPU: dodeca-core 64-bit kabylake
Clang: 16.0.0 build 1600
Git: 2.47.0 => /usr/local/bin/git
Curl: 8.7.1 => /usr/bin/curl
macOS: 15.1-x86_64
CLT: 16.1.0.0.1.1729049160
Xcode: 16.1

brew doctor

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed kegs have no formulae!
This means they were either deleted or installed manually.
You should find replacements for the following formulae:
  go@1.16

Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  go@1.16
  go@1.17
  go@1.18
  go@1.19
  go@1.20
  go@1.21

Verification

What were you trying to do (and why)?

Submit a PR to upgrade openjdk@8 to version 1.8.0-432.

What happened (include all command output)?

brew test --retry --verbose embulk failed because it selected the incorrect version of openjdk@21 (it should actually select openjdk@8 on the Intel architecture):

on_arm do
  depends_on "openjdk@21"
end
on_intel do
  depends_on "openjdk@8"
end
/usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.3.0/bin/bundle clean
  ==> Testing embulk
  ==> Downloading https://search.maven.org/remotecontent?filepath=org/jruby/jruby-complete/9.4.8.0/jruby-complete-9.4.8.0.jar
  ==> Downloading from https://repo1.maven.org/maven2/org/jruby/jruby-complete/9.4.8.0/jruby-complete-9.4.8.0.jar
  /usr/bin/env /usr/local/Homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.4.3-60-g745596b\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 14.7\)\ curl/8.7.1 --header Accept-Language:\ en --fail --retry 3 --remote-time --output /Users/brew/Library/Caches/Homebrew/downloads/db8894972d158eba6af122d132a624d09cfd39c16a8db6b30173a70899e18307--jruby-complete-9.4.8.0.jar.incomplete --location https://repo1.maven.org/maven2/org/jruby/jruby-complete/9.4.8.0/jruby-complete-9.4.8.0.jar
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed

    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  100 31.9M  100 31.9M    0     0   100M      0 --:--:-- --:--:-- --:--:--  100M
  ==> Verifying checksum for 'db8894972d158eba6af122d132a624d09cfd39c16a8db6b30173a70899e18307--jruby-complete-9.4.8.0.jar'
  cp -p /Users/brew/Library/Caches/Homebrew/downloads/db8894972d158eba6af122d132a624d09cfd39c16a8db6b30173a70899e18307--jruby-complete-9.4.8.0.jar /private/tmp/jruby-complete-20241101-7906-5s7uk4/jruby-complete-9.4.8.0.jar
  ==> /usr/local/Cellar/embulk/0.11.5/bin/embulk -X jruby=file:///private/tmp/embulk-test-20241101-7906-yw06o/jruby-complete-9.4.8.0.jar gem install embulk --version 0.11.5
  /usr/local/Cellar/embulk/0.11.5/bin/embulk: line 3: /usr/local/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home/bin/java: No such file or directory
  /usr/local/Cellar/embulk/0.11.5/bin/embulk: line 3: exec: /usr/local/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home/bin/java: cannot execute: No such file or directory
  ==> Testing embulk (again)
  cp -p /Users/brew/Library/Caches/Homebrew/downloads/db8894972d158eba6af122d132a624d09cfd39c16a8db6b30173a70899e18307--jruby-complete-9.4.8.0.jar /private/tmp/jruby-complete-20241101-8200-84vjdr/jruby-complete-9.4.8.0.jar
  ==> /usr/local/Cellar/embulk/0.11.5/bin/embulk -X jruby=file:///private/tmp/embulk-test-20241101-8200-lwwcgy/jruby-complete-9.4.8.0.jar gem install embulk --version 0.11.5
  /usr/local/Cellar/embulk/0.11.5/bin/embulk: line 3: /usr/local/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home/bin/java: No such file or directory
  /usr/local/Cellar/embulk/0.11.5/bin/embulk: line 3: exec: /usr/local/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home/bin/java: cannot execute: No such file or directory
  Error: embulk: failed
  An exception occurred within a child process:
    BuildError: Failed executing: /usr/local/Cellar/embulk/0.11.5/bin/embulk -X jruby=file:///private/tmp/embulk-test-20241101-8200-lwwcgy/jruby-complete-9.4.8.0.jar gem install embulk --version 0.11.5
  /usr/local/Homebrew/Library/Homebrew/formula.rb:3082:in `block in system'
  /usr/local/Homebrew/Library/Homebrew/formula.rb:3018:in `open'
  /usr/local/Homebrew/Library/Homebrew/formula.rb:3018:in `system'
  /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11630/lib/types/private/methods/call_validation.rb:270:in `bind_call'
  /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11630/lib/types/private/methods/call_validation.rb:270:in `validate_call'
  /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11630/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added'
  /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/e/embulk.rb:65:in `block in <class:Embulk>'
  /usr/local/Homebrew/Library/Homebrew/formula.rb:2817:in `block (3 levels) in run_test'
  /usr/local/Homebrew/Library/Homebrew/extend/kernel.rb:539:in `with_env'
  /usr/local/Homebrew/Library/Homebrew/formula.rb:2816:in `block (2 levels) in run_test'
  /usr/local/Homebrew/Library/Homebrew/formula.rb:1199:in `with_logging'
  /usr/local/Homebrew/Library/Homebrew/formula.rb:2815:in `block in run_test'
  /usr/local/Homebrew/Library/Homebrew/mktemp.rb:90:in `block in run'
  /usr/local/Homebrew/Library/Homebrew/mktemp.rb:90:in `chdir'
  /usr/local/Homebrew/Library/Homebrew/mktemp.rb:90:in `run'
  /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11630/lib/types/private/methods/call_validation.rb:270:in `bind_call'
  /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11630/lib/types/private/methods/call_validation.rb:270:in `validate_call'
  /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11630/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added'
  /usr/local/Homebrew/Library/Homebrew/formula.rb:3141:in `mktemp'
  /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11630/lib/types/private/methods/call_validation.rb:270:in `bind_call'
  /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11630/lib/types/private/methods/call_validation.rb:270:in `validate_call'
  /usr/local/Homebrew/Library/Homebrew/vendor/bundle/ruby/3.3.0/gems/sorbet-runtime-0.5.11630/lib/types/private/methods/_methods.rb:277:in `block in _on_method_added'
  /usr/local/Homebrew/Library/Homebrew/formula.rb:2809:in `run_test'
  /usr/local/Homebrew/Library/Homebrew/test.rb:48:in `block in <main>'
  /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.3.5/lib/ruby/3.3.0/timeout.rb:1[86](https://github.com/Homebrew/homebrew-core/actions/runs/11624571402/job/32374214285?pr=194719#step:3:87):in `block in timeout'
  /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.3.5/lib/ruby/3.3.0/timeout.rb:41:in `handle_timeout'
  /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/3.3.5/lib/ruby/3.3.0/timeout.rb:195:in `timeout'
  /usr/local/Homebrew/Library/Homebrew/test.rb:54:in `<main>'

What did you expect to happen?

Select openjdk@8 correctly (@@HOMEBREW_JAVA@@).

Or

bottle do
  sha256 cellar: :..., arm: "xxxxxxxxxx"
  sha256 cellar: :..., x86_64: "yyyyyyyyyy"
end

Step-by-step reproduction instructions (by running brew commands)

==> brew fetch --retry embulk
==> brew install --only-dependencies embulk
==> brew install embulk
==> brew install --only-dependencies embulk
==> brew linkage --test embulk
==> brew linkage --cached --test --strict embulk
==> brew install --only-dependencies --include-test embulk
==> brew test --retry --verbose embulk

For more detailed logs, see here.

calvinit commented 3 weeks ago

https://formulae.brew.sh/formula/embulk

It’s necessary to distinguish between different architectures here. QQ_1730446562652

cho-m commented 3 weeks ago

In theory, it should not need multiple bottles as the Java path replacement is done at install time. The bottle should be using a placeholder, e.g. @@HOMEBREW_JAVA@@.

The issue appears to be combining dependencies into its variations isn't handled correctly for API.

When using a local tap, it seems okay, e.g.

❯ brew info embulk --json=v2 --variations | jq '.formulae[].dependencies'
[
  "openjdk@21"
]

❯ arch -x86_64 brew info embulk --json=v2 --variations | jq '.formulae[].dependencies'
[
  "openjdk@8"
]
❯ brew info embulk --json=v2 --variations | jq '.formulae[].variations.arm64_sequoia'
null

❯ brew info embulk --json=v2 --variations | jq '.formulae[].variations.sequoia'
{
  "dependencies": [
    "openjdk@8"
  ]
}

❯ arch -x86_64 brew info embulk --json=v2 --variations | jq '.formulae[].variations.arm64_sequoia'
{
  "dependencies": [
    "openjdk@21"
  ]
}

❯ arch -x86_64 brew info embulk --json=v2 --variations | jq '.formulae[].variations.sequoia'
null

However, the API data is generated based on single platform:

❯ curl -sL "https://formulae.brew.sh/api/formula/embulk.json" | jq '.dependencies'
[
  "openjdk@21"
]
carlocab commented 3 weeks ago

However, the API data is generated based on single platform:

That's a bug, but I wonder why it affects is in CI when we set HOMEBREW_NO_INSTALL_FROM_API.

cho-m commented 3 weeks ago

As workaround, can just use same OpenJDK on embulk. I split dependency based on upstream status, but probably doesn't matter if we just use newer one.

For issue, could also be a general keg relocation problem. Would need to check on that code. The bottle itself has:

export JAVA_HOME="${JAVA_HOME:-@@HOMEBREW_JAVA@@}"
exec "${JAVA_HOME}/bin/java"  -jar "@@HOMEBREW_CELLAR@@/embulk/0.11.5/libexec/embulk-0.11.5.jar" "$@"

So as long as bottle pour substitutes right path then it should have worked.

EDIT: Another possibility is if it is reading tab from manifest.

cho-m commented 3 weeks ago

It is the manifest rather than JSON API. Sadly no idea how to handle this for accurate tab in all bottle. May need something similar to Perl's preferred option.

Another idea is skipping manifest's tab if OS/arch doesn't match, though this may not work if we remove info based on https://github.com/Homebrew/brew/issues/17395

Tab from https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fembulk/290843166

"sh.brew.tab": "{ ... {\"full_name\":\"openjdk@21\",\"version\":\"21.0.5\",\"revision\":0,\"pkg_version\":\"21.0.5\",\"declared_directly\":true}],