search

ember-a11y / ember-a11y-testing

A suite of accessibility tests that can be run within the Ember testing framework
MIT License
137 stars 49 forks source link

Question: custom config of middleware report? #512

Closed MelSumner closed 8 months ago

MelSumner commented 1 year ago

Looking at this part of the README: https://github.com/ember-a11y/ember-a11y-testing/tree/2b479c9e0fbea702cfde776372d194458ce07357#test-middleware

I'm able to successfully this up to run, and it returns a report with what looks like a (dasherized?) timestamp for the .json file name.

To take this to a more automated cron job fashion, I'd like to run this in a GITHUB action. I can mostly get it to work, but I'm just not quite sure how to get the name of the report produced by the middleware so I can save it as an artifact or even commit it to the repository in the ember-a11y-report folder in the same way as if I'd run the command in my CLI.

Example: After setting up the middleware, I ran the a11y audit on my codebase, and a file was added to the ember-a11y-report folder: 2023-06-28T16-40-05-049Z.json.

How would I get the timestamped file name returned in a GitHub action so it could be archived as an artifact? Any hints? I checked through the source code and the tests but I couldn't quite figure it out.

Also, I'm writing this as an issue because I'd like to add this information to the docs.

MelSumner commented 1 year ago

Since I was able to come up with a solution that worked, I'll open a PR that adds some documentation about setting things up to run in a GitHub action.

I wasn't able to resolve my original question (how to get the name of the file returned from the produced middleware report) but I was able to use a wildcard in my artifact path and that worked.

drewlee commented 1 year ago

I may be missing some context, but can't we just use a wildcard or glob pattern to specify the report? For example, ember-a11y-report/*.json or **/ember-a11y-report/*.json (depending on the project structure). This is how we handle artifact archiving in our internal apps.

scalvert commented 1 year ago

Hey @MelSumner, as @drewlee mentioned, using a glob is probably the best way to do this. If you need to persist this file via GitHub Actions, you can likely do a few things:

  1. If the a11y tests are run in a single job, you can simply commit that file and push those changes directly in your actions. It's a bit odd to be mutating the repo while running in an action, but if that's all the action is doing this seems reasonable.
  2. If the a11y tests are run across jobs, you would need to use the actions/upload-artifact action to store the file across jobs, and access it from another job via actions/download-artifact.

It'd also be possible to update the @scalvert/ember-setup-middleware-reporter to additionally take a config option that would allow you to specify your own file pattern. I'm not sure that's needed here, but is an option.

MelSumner commented 10 months ago

@drewlee @scalvert I must have been doing something silly because the wildcard glob wasn't working for me. Thank you both for responding!

acorncom commented 10 months ago

@MelSumner looks like we can probably close this issue at this point?