[CI] Make `CiMatrix` respect `depends_on macos:` when choosing runners

Logicer16 commented 3 years ago

[X] I understand that if I ignore these instructions, my issue may be closed without review.
[ ] I have retried my command with --force.
[ ] I ran brew update-reset && brew update and retried my command.
[ ] I ran brew doctor, fixed as many issues as possible and retried my command.
[X] I have checked the instructions for reporting bugs.
[X] I made doubly sure this is not a checksum does not match error.

Description of issue

depends_on macos: specifies the versions of macOS that are supported for a cask. When CiMatrix determines the runners a cask should be tested on without considering whether the runner actually is a macOS version supported the cask, causing checks to fail on environments that the user can't install/run the cask on themselves, and that the developer hasn't designed the cask for.

Required for Homebrew/homebrew-cask-versions#11231

Command that failed

brew ruby -- "$(brew --repository homebrew/cask)/cmd/lib/generate-matrix.rb" "https://api.github.com/repos/Homebrew/homebrew-cask-versions/pulls/11231"

Output of command with `--verbose --debug`

Unchanged from standard output: (https://github.com/Homebrew/homebrew-cask-versions/pull/11231/checks?check_run_id=2882107148)

Click to expand

[
  {
    "name": "syntax (ubuntu-latest)",
    "tap": "homebrew/cask-versions",
    "runner": "ubuntu-latest",
    "skip_audit": true
  },
  {
    "name": "test safari-technology-preview (macos-10.15)",
    "tap": "homebrew/cask-versions",
    "cask": {
      "token": "safari-technology-preview",
      "path": "./Casks/safari-technology-preview.rb"
    },
    "audit_args": [
      "--appcast",
      "--online"
    ],
    "skip_install": false,
    "runner": "macos-10.15"
  },
  {
    "name": "test safari-technology-preview (macos-11.0)",
    "tap": "homebrew/cask-versions",
    "cask": {
      "token": "safari-technology-preview",
      "path": "./Casks/safari-technology-preview.rb"
    },
    "audit_args": [
      "--appcast",
      "--online"
    ],
    "skip_install": false,
    "runner": "macos-11.0"
  }
]::set-output name=matrix::[{"name":"syntax (ubuntu-latest)","tap":"homebrew/cask-versions","runner":"ubuntu-latest","skip_audit":true},{"name":"test safari-technology-preview (macos-10.15)","tap":"homebrew/cask-versions","cask":{"token":"safari-technology-preview","path":"./Casks/safari-technology-preview.rb"},"audit_args":["--appcast","--online"],"skip_install":false,"runner":"macos-10.15"},{"name":"test safari-technology-preview (macos-11.0)","tap":"homebrew/cask-versions","cask":{"token":"safari-technology-preview","path":"./Casks/safari-technology-preview.rb"},"audit_args":["--appcast","--online"],"skip_install":false,"runner":"macos-11.0"}]

Output of `brew doctor --verbose`

N/A: CI environment

Example output: https://github.com/Homebrew/homebrew-cask-versions/pull/11231/checks?check_run_id=2882107148

Output of `brew tap`

N/A: CI environment

Example output: https://github.com/Homebrew/homebrew-cask-versions/pull/11231/checks?check_run_id=2882107148

carlocab commented 3 years ago

I'm not sure I understand the issue here. Your PR has

  if MacOS.version <= :big_sur
    version "126,071-57331-20210621-f40f3ff7-4076-4a0b-a86e-99ea6abbfb6a"
    sha256 "b9dabd73ff2e2069da43d436a911458dcee2eac1c25fa5c679a948f5f60c71ba"
  else
    version "126,001-66930-20210616-c91fc8a5-bfeb-47ec-80c2-0d54c6ebbcee"
    sha256 "d7547c77c709db6f551a5be1c60fcdff3b116579aa24038ebe2936003e8c3af4"
  end

which means that version 126,071-57331-20210621-f40f3ff7-4076-4a0b-a86e-99ea6abbfb6a is run on both Catalina and Big Sur. That seems to be reflected accurately in the output of generate-matrix.rb.

Could you elaborate on what you think the problem is?

Logicer16 commented 3 years ago

The PR also has

depends_on macos: ">= :big_sur"

requiring that it be run on Big Sur or later. The latest version does not support running on Catalina nor does it provide a valid installer that can be run on Catalina, causing unintended behaviour if the install or livecheck where to proceed on Catalina.

carlocab commented 3 years ago

Ah, yes, I missed that. That makes sense. Could you open a PR for this?

Logicer16 commented 3 years ago

Could you open a PR for this?

I would ... if I knew what I was doing. I could try and figure it out when I have some time, otherwise its open to whoever wants it.

carlocab commented 3 years ago

Sure, that's fine. Feel free to give us a shout if you get stuck. Even a draft PR that doesn't quite work fully would be a great start.

Homebrew / homebrew-cask