Integrate into web platform tests

[xi]

I just recently learned about https://web-platform-tests.org/ (wpt). I had to think about how we discussed the test format in https://github.com/matatk/landmarks/issues/303 and wonderd whether we were reinventing the wheel. Maybe these tests are better suited to be integrated into wpt. What do you think?

[matatk]

Hi @xi and sorry for my very slow reply; I have only just seen the notificatio of this message.

I agree it's worth looking into WPT, though I think that's more about checking that browsers/UAs are behaving/rendering content correctly, whereas I see things like our extensions more as assistive technologies. There is in fact a similar sort of project (I think) that is concerned with automating testing of assistive technologies (I found two links: a W3C Wiki page on automated testing and a GitHub repo on automated testing).

I've been meaning to research these in more detail, but have struggled to find the time. My intention was to develop this repo a bit more (possibly adding another tool to automate the testing of our extensions, and maybe NVDA) and then see if these tests could be integrated with one of those projects. But doing the research into WPT and ARIA-AT first, or in parallel sounds like a good idea. I hope to get back to this soon. If you have any more thoughts on them in the meantime, do let me know.

[matatk]

Just a quick update: things have been busy, but I have started researching this. I expect to more fully understand the different groups and projects relating to testing as early as possible in January, and will let you know what I have found then. Meanwhile, best wishes for the holiday period and new year.

[matatk]

Hello @xi. I have looked into this a bit more and wouldn't say I fully understand the different projects hehe, but I do have some findings and thoughts. I am lucky to have some colleagues who know about this stuff, and am in the process of asking their thoughts too. Anyway, here is what I found out about three testing-related projects.

Next steps:

• Please let me know what you think of the below.
• I'll post here as/when I find out more or get more input.
• If you think we should propose integration with either of the projects below, let's see if there are any things we need to address/fix/add here before we do that.

Testing-related projects

Accessibility Conformance Testing

The W3C's Accessibility Conformance Testing (ACT) project is about standardising the process of evaluating the accessibility of web content—so not applicable here.

Web Platform Tests

The Web Platform Tests project is a cross-community effort to write tests for the "Web-platform stack" which is basically the whole collection of technologies and APIs that browsers support. It is designed to run in a cross-browser way, with tests being run either via JS or via a Python web-driver set-up.

How it's a good fit

• The tests we want to do very-much relate to web content.
• We could make tests for browsers themselves.

How it may not be a good fit

• It's focused on testing the browser, not assistive technologies that either run inside the browser, or on top of it. Our use case may be considered out-of-scope.
• If we wanted to write a test to check that the browser had correctly scanned the landmarks (or headings) on a page, we'd need to use some external code. The README for the "core-aam" tests describes this process. This is needed because to test the browser, we would have to run some code outside of it. However it doesn't seem much progress has been made on the adapters—the W3C ARIA testing wiki page I mentioned above links to several adapter projects though on first glance they don't seem very active.
• Alternatively we could just make the tests manual (like the files in the "core-aam" folder as it is now). But if I wanted to use the test cases in the Landmarks extension's test suite, or in some other future tool that could hypothetically test a screen reader, I'd have to import the entire web-platform-tests repo, which seems over-the-top.

Assistive Technology ARIA Experience Assessment

The W3C's ARIA-AT project seeks to provide a test suite for assistive technologies (including screen readers) to show how they meet specs such as ARIA. The ARIA-AT Community Group's Call for Participation is from some time ago, but the GitHub repo remains active.

I gather their focus at the moment is on standardising manual testing, and the reporting of it, but they are considering how to automate testing in future. They have an issue/discussion on the Automation test format which links to a Google Doc that minutes a detailed discussion on the automated test formats.

Why it's a good fit

• It's still about web content.
• It's about testing assistive technologies, as opposed to browsers (though we may think it'd be nice to test both of these at the same time).

Why it may not be a good fit

• It doesn't look like the automated side of things has been standardised yet, though this is good, as it gives us an opportunity to propose things.
• Again there may be the problem of having to import a large repo if all you want is the landmarks/headings test cases.

[xi]

Thanks for the writeup, that was extremely informative!

I believe it makes sense to think about both the sort term and the long term.

• In the long term it probably makes sense to integrate into any (or all) of the existing test suites. From the information that you have gathered it sounds like ARIA-AT is the best fit, but it might be useful to integrate these cases into more than one suite.
• In the short term, none of these seems to provide the required level of automation. So in the short term integration is not feasible.

Even if these suites are not automated themselves, they still provide a collection of test cases that can be automated by separated projects. For aria-api I created a script that converts WPT cases to automated tests.

[matatk]

@xi: sorry for my very slow reply; I misread the notification and thought I was waiting for a reply from you—apologies.

I agree with your thoughts on the current state of things, and that it would be good to get these tests into an established framework in the long term. (I'm hoping we'll be able to find a way to import the tests without lots of other dependencies, but that is something we can keep in mind whilst working towards adoption into a larger project, and I suppose it would most likely only be a dev-time dependency.)

I also agree ARIA-AT sounds good, and your script is very cool :-). I also see from the wider commit why you wrote it, it's really neat that you were able to use the existing set of tests.

I've been meaning to ask: would you like commit access to this repo?

[xi]

I've been meaning to ask: would you like commit access to this repo?

I am not sure. I really like to exchange ideas with you. But I don't think I have a clear enough vision for this repo to comfortably know what to merge. For now I think I would prefer not to have commit access. If I have ideas for changes I can still create a pull request. Also, feel free to mention me if you want feedback on anything, I am happy to have a look.

[matatk]

@xi will do, and thanks for all of your helpful feedback so far :-).