search

bocoup / aria-at-gh-actions-helper

A collection of workflows for automation process of aria-at.w3c.org
0 stars 2 forks source link

Support runs with specific AtVersions #21

Closed stalgiag closed 3 months ago

stalgiag commented 4 months ago

Overview

This PR adds support for specifying the AT version that a runner should use.

Notes

jugglinmike commented 3 months ago
  • I opted for using conditionals in every action for the early exit so that that part of the matrix shows as completed rather than errored.

I believe the only people who will see this are folks who are debugging the workflow. If that's the case, wouldn't the "error" status make it a little easier to recognize which of the two jobs is valid?

It is verbose and I am open to alternatives if anyone has a suggestion.

If the "error" status doesn't provide meaningful job differentiation, then maybe we could address the verbosity by reverting to your original exit 1 approach and adding continue-on-error.

stalgiag commented 3 months ago

I'd be okay with doing that and you are right that continue-on-error would allow the non-failed job to continue running. The reason I opted for no error at all is that the entire job shows as "errored" when viewed from certain pages since Github considers the run to have an overall status of error if any job in the run fails. This means that on certain views the information conveyed would be incorrect for our purposes. For example, this view would have red x's to the left of every run. Additionally the top-level view of the runs would show a red X. One would only see that any job in the run passed after clicking down further to show the status for the individual jobs. I agree that it isn't ideal to show the unused path as passed but it feels useful to be able to separate actual failures from expected behavior visually on the main view of all runs. I wish there was a way to exit early with a status other than error 🤔

I'll defer to the reviewers on this one!

jugglinmike commented 3 months ago

I'd be okay with doing that and you are right that continue-on-error would allow the non-failed job to continue running.

I realize that I don't actually understand what that feature does. I suggested it based on the documentation:

Prevents a workflow run from failing when a job fails. Set to true to allow a workflow run to pass when this job fails.

That made me think we could enable it, remove all the conditions, and observe the same behavior.

The trouble is, the way you've described it sounds much more intuitive (given the name of the feature) and suggests it will not allow us to simplify.

I guess there are two questions, here:

Do you know the answers to these questions offhand? Or are you set up to check experimentally?

The reason I opted for no error at all is that the entire job shows as "errored" when viewed from certain pages since Github considers the run to have an overall status of error if any job in the run fails. This means that on certain views the information conveyed would be incorrect for our purposes. For example, this view would have red x's to the left of every run. Additionally the top-level view of the runs would show a red X. One would only see that any job in the run passed after clicking down further to show the status for the individual jobs. I agree that it isn't ideal to show the unused path as passed but it feels useful to be able to separate actual failures from expected behavior visually on the main view of all runs.

Great point! Let's reserve errors for exceptional circumstances.

I wish there was a way to exit early with a status other than error 🤔

As long as we're making wishes, let's ask the genie to not create useless jobs in the first place :)

stalgiag commented 3 months ago
  • Does the "continue" in continue-on-error mean "continue [executing the subsequent steps in the job]" or "continue [executing the subsequent jobs in the workflow]"?

It means the latter with an error status assigned to this job. Other jobs still run.

  • Is the error propagated to the level of the workflow's status?

Yes. The job's have separate statuses but it appears that (at least for the purposes of display on Github) the workflow's status is calculated with something to the effect of if any of the workflow.jobs.status === ERROR then workflow.status = ERROR

As long as we're making wishes, let's ask the genie to not create useless jobs in the first place :)

You are much better at choosing wishes for the genie haha.

jugglinmike commented 3 months ago

So continue-on-error really has nothing to offer us here, after all. Too bad! This seems like the best we can do given the constraints.

You are much better at choosing wishes for the genie haha.

You're kind for saying so, but clearly I could have done a better job with my three.