Discussion: Improvements for the accessibility testing process

[annezazu]

What problem does this address?

Currently, no block themes are listed as accessibility ready and, in the process of trying to determine what would make them so, folks working to evaluate the readiness of Twenty Twenty-Two became blocked based on the current testing process for blocks in general. As a result, this issue seeks to take a step back to the initial pain points around accessibility testing for Gutenberg in order to then unblock the following work:

• Identifying current accessibility problems in blocks.
• Identifying any unique problems as they relate to block themes.
• Determining future guidelines for block themes.

In theory, if these pieces can fit together, block themes can be more accessible by default thanks to updates to blocks themselves with a few guidelines/best practices in place for theme authors to follow.

What is your proposed solution?

Based on various discussions there are a few ideas thrown out to move this forward and create better pathways for testing accessibility and improving the built in accessibility of block themes:

Publicly accessible index page containing all core blocks in their default state

To quote @joedolson:

“One of the problems is just being able to consistently catalog all of the core blocks. The most useful thing for me, and I think for the accessibility team as a whole, would be a publicly accessible index page containing a single instance of all core blocks in their default state. For testing purposes, they should have a basic level of content (e.g., images should have alt text included, galleries should have captions, menus should have items in multiple levels, etc.) Something like gutenberg.run, but focused on the front-end rather than the back-end, so that we can more easily test the output of blocks. Right now, the labor involved in setting up a site, building Gutenberg, finding the right block, etc., is a major pain point when all you really want to do is verify that the output of a block is OK.”

To add to this, in talking with @carolinan, this could be nicely combined with a style guide that shows what each block looks like by default.

Update theme unit test data

The information in place here currently is years out of date making it hard to rely on since it’s likely that the blocks aren’t reflected properly and/or are missing entirely, since loads of new blocks were created for full site editing. In particular, blocks not typically used in content would need to be included like the navigation block, comments blocks, search block, site title block, etc.

Props @joedolson for this idea.

Update Gutenberg’s Demo content to show all blocks/create another demo page for testing

When you activate the Gutenberg plugin, under Gutenberg there is a section called “Demo” that displays demo content. This could be updated to show all blocks allowing for streamlined testing without needing to manually add blocks. Alternatively, if this interferes with the user focused angle of the demo content, a second “demo” section could be included for development that lists all blocks. Ideally, it would be nice too to be able to choose which block or category of blocks you want displayed to make it easier to test against.

To go a step further, there could be a very simple plugin that provides this information/functionality if altering the demo content is a no go.

Props @carolinan for this idea.

What other ideas do you have? Of these, which feels most viable and high impact to move forward right now with current project resources?

To state the only other obvious item, I think someone with deep experience across block themes, Gutenberg, and accessibility might need to manually dig into this problem by looking at a theme like Twenty Twenty-Two in order to figure out what's needed to improve the accessibility of blocks and how theme guidelines will need to evolve in a block theme world.

Thank you to @carolinan and @joedolson for talking about this and collaborating on this issue.

[alexstine]

@talldan told me not too long ago that Gutenberg was using Storybook at least for components. I wonder if this could help us if it was more visible to contributors?

[carolinan]

The problem with manually updating the theme test data, is that "someone" would need to keep updating it every two weeks when Gutenberg is updated, since both markup and classes keeps changing (especially for the navigation block). We need a low effort, automated solution.

[joedolson]

This is primarily a problem around automation: testing front-end content, whether you're creating a new theme, assessing the accessibility of a theme, or assessing the pieces that might be used by a theme, depends on being able to quickly collect together all the various pieces that could potentially be used.

This is critically different from the development process, which is largely component driven: e.g., you're working on the navigation component, you have a build in process with content for that component, you test as you go, etc. But theme development is about potential usages, so what you need is broad coverage of those possibilities without the need to spend hours generating each block by hand or configuring the theme.

The theme unit test data has always been about creating content that can be used to test themes: see if all of the required classes are present and styled, see whether various types of content are handled as expected, etc. Because WordPress output was stable for many years, this didn't need to be adjusted frequently, and was very maintainable.

We need a similar way to handle blocks - but since they are extremely unstable, we can't keep doing it the same way; block output needs to be available through automation.

[carolinan]

Each block already has a test fixture, correct? How can we take advantage of that? Link to fixtures folder.

[jffng]

Each block already has a test fixture, correct? How can we take advantage of that?

Publicly accessible index page containing all core blocks in their default state

Could we leverage the fixtures (the .html file in particular) to generate this page containing all the core blocks?

We would probably need to maintain a list of which blocks / fixtures to output, because some are deprecated or may not make sense to output in isolation (e.g. core__comments-pagination-next.html). But this seems like less effort long term than having to manually update an XML every two weeks.

[joedolson]

Any block that's potentially going to be part of an FSE theme should be included; if it's already included as sub-content of another block, then including it without that context probably doesn't make sense, but things like comments pagination are definitely worth including, generally.

Deprecated blocks all have that marked in the fixture file names, so can't those just be eliminated by a string verification? It seems highly unlikely we'd ever have a block that included 'deprecated' for any other reason. That would be terrible... ;)

[mikachan]

I've been thinking about how we could use Storybook to help with this, as @alexstine mentioned. It would cover a lot of what we need, including a single page with all blocks and a way to integrate with tools like jest-axe and the Storybook a11y add-on.

However, I'm not sure it's the best solution here, as I'm not sure how Storybook would interpret the blocks. Although there is a plain HTML solution, maybe we could use the fixtures to build stories? Or we could generate a story file at the same time as each fixture?

If Storybook doesn't sound like a viable solution, then I really like @jffng's idea of building a single page of blocks from the fixtures. We could then use this page as a basis for automated and manual accessibility testing. Maybe we could hook into the fixtures:generate command to continually add new blocks to the page.

[matiasbenedetto]

:wave: I submitted a PR trying to implement the nice ideas you discussed in this issue I would love to have your feedback :smile_cat:

[mikachan]

I've also just submitted a PR using the Storybook idea. I'm not convinced, but it was fun trying it out 😅 If anyone thinks this is a good solution, I'm happy to put more time into it.

@matiasbenedetto's PR looks like a great idea too 👍

[mirka]

Could @youknowriad's BlockBook be a perfect fit for this? It's basically a Storybook for Gutenberg blocks.

[joedolson]

@mirka That does look like it has the pieces we were looking for - seems promising! Glad to see all this ideas coming out of the wood work here!

[mikachan]

Could @youknowriad's BlockBook be a perfect fit for this?

Yeah this looks ideal! This is basically what I was trying to aim towards. Awesome! We should give this a spin with some automated tools (I'm happy to try this, but also happy if anyone beats me to it.)

[skorasaurus]

Regarding Update Gutenberg’s Demo content to show all blocks/create another demo page for testing:

I run a customized version of block-unit-test ; the plugin, when activated, creates a draft page with all of the blocks and block content that you specify in the code.

It's definitely a limited solution because you have to continually add newer versions of the blocks when there's new block markup that appears.

[tellthemachines]

The advantage of Storybook is it comes with inbuilt a11y testing; of course that can also be added to BlockBook, especially if it becomes a WordPress package 🙂

Regarding fixtures, I don't think they're the best way forward here; I left a comment on #39437 with a bit more context on that. E2e would be better suited to run comprehensive automated tests against block markup.

If I understand the problem correctly, we want to:

• test core block output for potential a11y issues, and

• provide a way of testing theme a11y against all core blocks

For the first part, Storybook/BlockBook and/or e2e tests will work.

For the second part, ideally we'd have a tool that generates either a post or a template with all the core blocks in it (we'd probably also want all block variations and major options, such as mobile navigation).

The problem with manually updating the theme test data, is that "someone" would need to keep updating it every two weeks when Gutenberg is updated, since both markup and classes keeps changing (especially for the navigation block)

I don't think we should be generating markup from the Gutenberg plugin, but from the latest version of WP. Things in the plugin change frequently because we're still working them out 🙂 , and the last months have been a bit fuzzy with FSE being in the process of migrating to Core, but from the point it's stable, we'll expect theme authors to work with what's in Core and not in Gutenberg. Any tools we develop now should reflect that expectation.

Another thing to take into account for this block generation tool is environment-dependent output: e.g. navigation links to posts won't work correctly unless they are generated from the actual environment content.

Could we maybe build a plugin that creates a post and goes through all the buttons in the global inserter, adding in a copy of each block?

[carolinan]

I want to highlight that from conversations with theme developers, and from seeing the themes that are submitted, this "we'll expect theme authors to work with what's in Core and not in Gutenberg" is not what is actually happening.

[tellthemachines]

I want to highlight that from conversations with theme developers, and from seeing the themes that are submitted, this "we'll expect theme authors to work with what's in Core and not in Gutenberg" is not what is actually happening.

It may not be what's happening now, because the site editor is still in beta, and theme authors need to work to what's coming instead of what is already in Core, but once it becomes stable we'll go back to our usual process. Theme authors should not have to follow the development process; they should have a stable API in Core they can work with, with good documentation on best practices etc.. What we're going through currently is very much exceptional circumstances 😅

[carolinan]

Could we maybe build a plugin that creates a post and goes through all the buttons in the global inserter, adding in a copy of each block?

Maybe a stupid question, but can't Playwright do that?

• Login
• Create new page
• Add one of each block
• Save
• Go to front

But how can that page content then be stored for both automated and manual testing?

[tellthemachines]

Not stupid at all! An e2e framework could be used to generate content and save it as a post. Run it in whatever instance we're using to test, and we have content ready to go. It should be a separate script from our actual e2e tests though, and preferably contained in a plugin that can be installed on the site we want to test themes on.

[matiasbenedetto]

:wave: I just want to point out that the https://github.com/WordPress/gutenberg/pull/39437 PR implements what you are proposing in the latest comments using the e2e infrastructure that exists in Gutenberg (Puppeteer + Jest Puppeteer + @wordpress/jest-puppeteer-axe ). @carolinan @tellthemachines

Maybe a stupid question, but can't Playwright do that?

• Login
• Create new page
• Add one of each block
• Save
• Go to front

I copy from the PR description:

The steps involved are in the automated testing process are:

1. Spin up a WordPress instance
2. Activate the latest WordPress default theme (twentytwentytwo)
3. For each block definition:
   • Checks if there are valid fixtures for that block :new:
   • Gets all the valid fixtures available
     • For each valid fixture creates a post containing only that block
     • Navigate that post url
     • Run a11y tests over each rendered block using axe. (If there are specific a11y test options set in block.json they are followed.)
4. Report results

[carolinan]

The difference as I understood from the comments on that PR is that the above relies on fixtures, where the block may be empty.

[tellthemachines]

I just want to point out that the https://github.com/WordPress/gutenberg/pull/39437 PR implements what you are proposing in the latest comments using the e2e infrastructure that exists in Gutenberg

Independently of how we write the tests, we need to work out the best way of making a bunch of content, with examples of all the core blocks, available for theme authors to test.

It would be great if whatever automated tests we create can be run on any theme, as that will help the review task, but there will always be a manual testing element to the review process that requires some dummy content, so automation won't cover everything.

The difference as I understood from the comments on that PR is that the above relies on fixtures, where the block may be empty.

I left another comment on the above PR expanding on my reasoning.

[matiasbenedetto]

Based on your latest comments and suggestions I started a quick proof of concept around a package to test the blocks using the E2E tools.

Basically, it is a collection of e2e tests that uses @wordpress/e2e-test-utils and @wordpress/jest-puppeteer-axe and adds each block with the necessary mock data/assets, instead of fixtures, to have meaningful content to run the a11y test.

I would love to have your input on this latest approach.

[carolinan]

I think that if possible, we should try to avoid having one test file for each block, as that is part of the maintenance problem.

[joedolson]

If the data/assets test files can be checked as part of the acceptance process - e.g., changes to blocks need to confirm that the test assets have been updated, and that can be automated, then I think this is doable. It's just that it won't work as a secondary process, where running the tests first requires manually verifying that each test is still current.

Would you agree with that, @carolinan?

[carolinan]

I feel like we are talking about two different things, and I believe we need to solve both and that we need to be a bit more specific;

• A set of e2e tests included in Gutenberg
• A tool that theme developers can use to test the blocks in their themes (and I suppose, plugin developers too)

For tests included in Gutenberg, we do need to use Playwright: https://make.wordpress.org/core/2022/03/23/migrating-wordpress-e2e-tests-to-playwright/

https://github.com/WordPress/gutenberg/issues/38851

[joedolson]

It does seem like this is really addressing two completely separate issues, I agree. They both need to be addressed, but the primary issue that this discussion is intended to be about is the tool that allows block output to be tested by people.

Things like including axecore as an automated test is fine, but does not meet the needs of theme developers or accessibility testers to actually assess the quality of the output, since it's far too limited in scope.

Do we need to split this into two separate discussions for better clarity?

[carolinan]

We now have two separate tools, how do we move forward? @matiasbenedetto https://github.com/matiasbenedetto/a11y-blocks-test @JustinyAhin https://github.com/JustinyAhin/generated-blocks

[joedolson]

@JustinyAhin's tool is providing the front-end view that I was looking for, so that's a great start. It needs some work to make it really useful for testing: e.g., pre-populating the blocks with content suitable for testing. E.g., the comment block doesn't provide a great test case if it just indicates "no comments", ensuring the tag cloud is populated with tags, etc.

Justin - would you like these types of comments raised as issues on your repo?

[JustinyAhin]

@joedolson it would be great to raise the comments as issues on the repo, yes.

As for adding more content on the blocks, yes, definitely a good idea. I am also discussing with @carolinan about having each block on a separate page.

[JustinyAhin]

@matiasbenedetto I really like how you structure your repo. I see that you are using Puppeteer. Have you considered using Playwright, since Gutenberg (and soon Core) tests are being migrated to that? https://github.com/WordPress/gutenberg/tree/trunk/test/e2e

[matiasbenedetto]

Yep, I was planning to move to Playwright since the e2e are being migrated from Puppeteer to Playwright, but I saw that @JustinyAhin has already written the tests using that. After the last week's additions, the repos are doing basically the same, so it doesn't make sense, to me, to duplicate the effort here.

Apart from that, I think it would be nice to reuse the Gutenberg e2e-test-utils-playwright to avoid code duplication and the need to maintain 2 sets of utils, but I guess the package is not ready yet.

[annezazu]

Following up here, I stumbled on https://github.com/abstractwp/block-unit-test from a Gutenberg Times newsletter (shout out to @bph) and noticed that this might solve what's being asked for here. @joedolson mind taking a look?

Testing every core block — and every variation of every block — is no small task. That's why I built the Block Unit Test WordPress plugin. Deploy the Block Unit Test WordPress plugin and review every core Gutenberg block to ensure your theme fully supports Gutenberg.

[carolinan]

That will not work because the markup is hard coded in the plugin files. Meaning when there is a change to a block, the block in the plugin is no longer correct.