alphagov / accessibility-tool-audit

Automated accessibility tools audit
https://alphagov.github.io/accessibility-tool-audit/
MIT License
80 stars 30 forks source link

axe 2021 audit update #55

Open DiedeGu opened 3 years ago

DiedeGu commented 3 years ago

Hi @selfthinker,

Here are some updates for 2021.

AXE:As I worked through these changes I saw that axe seems to have removed checks that caused errors before. I can not judge if this is a case of regression or that the need to catch these problems for accessibility has changed. Which could be possible in cases where elements are deprecated or in the more grey areas. For instance layout done with tables is still possible but is less and less in actual use.

Since the changes on axe alone are minimal it is also possible to double check my work, I might have made a mistake.

Kind Regards,

selfthinker commented 3 years ago

Thanks @DiedeGu. I finally managed to slowly look through these.

I'm happy with most of this but found 4 instances where I see other results. Please let me know if you disagree with any of these.

Differing results

Content is not organised into well-defined groups or chunks, using headings, lists, and other visual mechanisms

You suggest a change from 'notfound' to 'error'.

The issues axe lists are:

Document must have one main landmark All page content must be contained by landmarks (11x)

I wouldn't count any of them as the error we were looking for. Although landmarks help break up the text, they are only for screen reader users, and in this case they wouldn't make any sense.

Table that only has TH elements in it

You suggest a change from 'error' to 'notfound'.

axe lists this to review:

All th elements and elements with role=columnheader/rowheader must have data cells they describe

As it is listed to review I would usually change it to 'manual'. But in this case it is phrased in a way which makes it clear something is wrong. So, keeping 'error' seems more appropriate.

Image link with no alternative text

You suggest a change from 'error' to 'notfound'.

But axe lists it as an error:

Links must have discernible text

Object not embedded accessibly - wmode parameter not set to window

You suggest a change from 'notfound' to 'error'.

But the only issues axe lists are these:

<object> elements must have alternate text Elements must have sufficient color contrast All page content must be contained by landmarks

None of those point to the issue with the wmode parameter.

Warning vs manual

There are two more cases which might come from a misunderstanding of how to use a 'warning' vs a 'manual' status.

You suggest to change both of them to 'warning'. But the issue text makes it clear that it needs to be reviewed and that axe doesn't know if something is wrong or not. That's why I would change it to 'manual' instead.

DiedeGu commented 3 years ago

Hi @selfthinker,

Currently it has been a bit busy at work so I did not find the time yet to read your response, I hope to do so next week, and then send in the other tools I audited after that

DiedeGu commented 3 years ago

Hi @selfthinker

Sorry for the long, long wait before responding, things kind of picked up at work so I didn't find the time to review your remarks. The short summary is that I agree with all four of your findings and I have just been a bit to careless in jotting down the results. I did all the tests with all the tools you could install as chrome extensions in one sitting so I reckon I just simple overlooked stuff, misread results or looked at the wrong tab.

For the first and fourth remake I think I misread the output and the second and third remark I think I just tested the wrong page.

In the area of warnings and manual, I would like to ask you to be the final judge. I defined warning as any notice by the tools that proms me to do a manual check. anything I labelled as manual are the "manual" tests that are given in the tools that are shown on default for any page. Examples of the latter are found in siteImprove which gives a single one for instance and in HTML codeSniffer gives 42 on default on any page.

Take all the time you need with the further proces and let me know when I can push the other changes. I will be busy for the coming months but I do intent to send you all the work I did so far.