Detect step-level continue-on-error and treat it as a failed CI

[eregon]

From https://github.com/oracle/truffleruby/issues/1624 https://github.com/sporkmonger/addressable/runs/2977469877?check_suite_focus=true looks green because step-level continue-on-error is used: https://github.com/sporkmonger/addressable/blob/24336385de0261571b3adaad0431459edb420c79/.github/workflows/test.yml#L150

It would be very useful to be able to detect such a build as failing. Maybe from annotations: https://github.com/sporkmonger/addressable/actions/runs/995606626, or there might be an API to query the outcome of specific steps.

https://github.com/actions/runner/issues/2347 might be helpful.

[bjfish]

I don't see any field in the API that indicates failure for the step that failed: https://api.github.com/repos/sporkmonger/addressable/actions/jobs/2977469877

[eregon]

Indeed, it seems outcome is not in the API, only conclusion, unfortunate. https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#steps-context has outcome. Might be worth filing a feature request to GitHub.

https://api.github.com/repos/sporkmonger/addressable/actions/jobs/2977469877 6
name	"Run specs"
status	"completed"
conclusion	"success"
number	7
started_at	"2021-07-03T04:28:41.000Z"
completed_at	"2021-07-03T04:31:29.000Z"

---

So probably the next best thing is to use the "Checks" API. https://api.github.com/repos/sporkmonger/addressable/check-runs/2977469877 has

output
title	"test (truffleruby-head, ubuntu-18.04, Gemfile)"
summary	"There are 1 failures, 0 warnings, and 0 notices."
text	null
annotations_count	1
annotations_url	"https://api.github.com/repos/sporkmonger/addressable/check-runs/2977469877/annotations"
name	"test (truffleruby-head, ubuntu-18.04, Gemfile)"

Which I found via https://api.github.com/repos/sporkmonger/addressable/actions/runs/995606626 check_suite_url https://api.github.com/repos/sporkmonger/addressable/check-suites/3148548132 check_runs_url https://api.github.com/repos/sporkmonger/addressable/check-suites/3148548132/check-runs 15 url https://api.github.com/repos/sporkmonger/addressable/check-runs/2977469877

but there might be a more direct way.

[eregon]

Actually https://api.github.com/repos/sporkmonger/addressable/actions/jobs/2977469877 (which you posted) has check_run_url -> https://api.github.com/repos/sporkmonger/addressable/check-runs/2977469877

[eregon]

And https://api.github.com/repos/sporkmonger/addressable/actions/runs/995606626/jobs also has the check_run_url directly.

[bjfish]

I implemented this at: https://github.com/eregon/truffleruby-gem-tracker/commit/13a538a21733fa19c257bc0c8af50626d711d709

Before:

 % bin/gem_tracker status addressable --aggregate=false
addressable ✗ 21-07-2021 test (truffleruby, ubuntu-18.04, Gemfile) https://github.com/sporkmonger/addressable/runs/3128232442
addressable ✓ 21-07-2021 test (truffleruby-head, ubuntu-18.04, Gemfile) https://github.com/sporkmonger/addressable/runs/3128232461
addressable ✗ 21-07-2021 test (macos-10.15, truffleruby)          https://github.com/sporkmonger/addressable/runs/3128232689

After:

% bin/gem_tracker status addressable --aggregate=false
addressable ✗ 21-07-2021 test (truffleruby, ubuntu-18.04, Gemfile) https://github.com/sporkmonger/addressable/runs/3128232442
addressable ✗ 21-07-2021 test (truffleruby-head, ubuntu-18.04, Gemfile) https://github.com/sporkmonger/addressable/runs/3128232461
addressable ✗ 21-07-2021 test (macos-10.15, truffleruby)          https://github.com/sporkmonger/addressable/runs/3128232689
Failing CIs: sporkmonger/addressable