livecheck error: undefined method `[]' for nil:NilClass

[fxcoudert]

See this PR: https://github.com/Homebrew/homebrew-core/pull/101026 which fails with:

2022-05-07T14:12:52.5809940Z ==> Running Formulae#formula!(kim-api)
2022-05-07T14:12:52.5962190Z ==> brew install curl
2022-05-07T14:13:18.9566470Z ==> Determining dependencies...
2022-05-07T14:13:20.0159040Z ==> brew fetch --retry bison cmake doxygen gcc gmp isl libmpc mpfr pkg-config
2022-05-07T14:13:25.8291340Z ==> Starting build of kim-api
2022-05-07T14:13:25.8291850Z ==> brew fetch --retry kim-api --build-bottle --force
2022-05-07T14:13:27.5416150Z ==> brew install --only-dependencies --verbose --build-bottle kim-api
2022-05-07T14:13:44.0953390Z ==> brew install --verbose --build-bottle kim-api
2022-05-07T14:14:01.5151110Z ==> brew livecheck --json --full-name kim-api
2022-05-07T14:14:02.1935240Z ==> brew uninstall --force bison cmake doxygen gcc gmp isl libmpc mpfr pkg-config
2022-05-07T14:14:03.4336150Z Error: undefined method `[]' for nil:NilClass
2022-05-07T14:14:03.4336730Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/lib/tests/formulae.rb:229:in `livecheck'
2022-05-07T14:14:03.4338360Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/lib/tests/formulae.rb:356:in `formula!'
2022-05-07T14:14:03.4338980Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/lib/tests/formulae.rb:18:in `block in run!'
2022-05-07T14:14:03.4339510Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/lib/tests/formulae.rb:17:in `each'
2022-05-07T14:14:03.4339990Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/lib/tests/formulae.rb:17:in `run!'
2022-05-07T14:14:03.4340510Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/lib/test_runner.rb:209:in `run_tests'
2022-05-07T14:14:03.4341060Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/lib/test_runner.rb:61:in `block in run!'
2022-05-07T14:14:03.4341580Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/lib/test_runner.rb:49:in `each'
2022-05-07T14:14:03.4342020Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/lib/test_runner.rb:49:in `run!'
2022-05-07T14:14:03.4342540Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/lib/test_bot.rb:126:in `run!'
2022-05-07T14:14:03.4343040Z /opt/homebrew/Library/Taps/homebrew/homebrew-test-bot/cmd/test-bot.rb:101:in `test_bot'
2022-05-07T14:14:03.4343500Z /opt/homebrew/Library/Homebrew/brew.rb:110:in `<main>'
2022-05-07T14:14:03.4387770Z ##[error]Process completed with exit code 1.

I do not know why we get undefined method[]' for nil:NilClass`, the livecheck block seems ok to me.

[samford]

I’m away for the next few hours but I’ll look into this when I return.

[cho-m]

First, I think this happens when livecheck doesn't match anything, which could be due to URL or pattern.

The actual error is due to how test-bot handles the result in https://github.com/Homebrew/homebrew-test-bot/blob/b4fd3be21fd543587731eac424e2bed41aa5cbf3/lib/tests/formulae.rb#L226-L229

        return unless livecheck_step.output?

        livecheck_info = JSON.parse(livecheck_step.output)
        return unless livecheck_info.first["version"]["newer_than_upstream"]

It probably isn't detecting the livecheck output correctly and fails the entire CI run rather than returning.

In the case there is a failure, livecheck_step.output is a valid JSON, but doesn't have version which would cause nil output.

Probably need to do some extra sanitization checks to make sure of things like livecheck_info.first["version"] is not nil.

---

On side note, in case of kim-api, livecheck result would probably be due to same reason as why download is failing. This may be fixed by changing download URL to

url "https://s3.openkim.org/kim-api/kim-api-2.3.0.txz", using: :homebrew_curl

However, the test-bot should also be fixed to avoid blocking CI runs..

[fxcoudert]

The error was with :homebrew_curl on the main URL. I tried to add it to livecheck's URL, but this is rejected by audit: https://github.com/Homebrew/homebrew-core/runs/6335695332?check_suite_focus=true

[Bo98]

Probably need to do some extra sanitization checks to make sure of things like livecheck_info.first["version"] is not nil.

The assumption was that the exit code would indicate success. It seems this is not always the case for --json so logic would need to be there to check status in the JSON but still fail overall CI in the case of a error.

[samford]

I tried to add it to livecheck's URL, but this is rejected by audit

livecheck block URLs don't accept trailing options like using: :homebrew_curl yet. This general idea is part of the forthcoming options feature in livecheck, which I still need to create a PR for (I've had it implemented for ages but it will involve some discussion and probably some rework). I've been busy with other things (e.g., tying up loose ends around my Utils::Curl changes and working on some refactoring) and there are a few other livecheck changes that I want to merge first but we'll eventually be able to explicitly specify that a livecheck block should use brewed curl.

In the interim time, livecheck will use brewed curl for the check if the stable (or head) URL includes using: :homebrew_curl. It looks like there was one revision where that was the case. Did that work when you tried it?

If not, kim-api may be another case where the check works fine locally but fails on CI. We encountered this scenario before with juman but I couldn't figure out why it only failed on CI, so the livecheck block was removed for the moment (despite it working locally).

The test-bot bug will be addressed in Homebrew/homebrew-test-bot#802 but that only appears because the livecheck block encounters an error, so we'll need to fix the livecheck block regardless.

[fxcoudert]

Even with using: :homebrew_curl in the main URL, it still fails. See https://github.com/Homebrew/homebrew-core/runs/6339592998?check_suite_focus=true