No definition of what constitutes an A, AA, or AAA conformance Level

[Helixopp]

There does not appear to be any clear definition of what parameters a success criteria must meet in order to qualify for any particular conformance level.

In order to be considered Conformance Level A does a success criteria have meet a certain level of severity, as in if not met it will severely adversely impact the user.

Likewise is an AA Conformance Level less severe/impactful? And so on.

Or is it that Level A Conformance is attributed to easy fixes, slightly more complicated issues are deemed AA Level, etc.?

Accessibility consulting companies are assigning severity levels to WCAG Success Criteria (critical, moderate, serious, etc.)which seem to be largely based on conformance level.

This is extremely subjective, and dangerously misleading. Lack of transparency of what criteria are required to be met for each conformance level is partly to blame.

If such information does exist it is too difficult to find.

[patrickhlauke]

@JAWS-test because that definition you link to defines the considerations/axes, but then does not actually define what A, AA, AAA are. it explains the considerations, but then misses the last bit that says how those considerations then map (even at a very broad/handwavy way) to A, AA, AAA. and i'm still surprised that some people are seemingly trying to gaslight folks about it

I mean, even just a simple statement at the end of that definition that says "consider the Levels to be rough groupings of similar SCs (similar based on these various axes/considerations). these levels don't necessarily imply importance or severity or any kind of hierarchy, on their own"

that would still not be satisfying (and arguably still not quite true), but at least it would provide that one missing bit from the definition.

[electronicwoft]

"why is this SC in level FooBar?" "because the group decided that it should go in FooBar" "so what is the meaning of level FooBar?" "it's the level that the group decided SCs like this one should be in"

it keep reminding me of: “When I use a word,’ Humpty Dumpty said in rather a scornful tone, ‘it means just what I choose it to mean — neither more nor less.’"

and to be clear, we're not asking for a definitive formula here, but clearly, the parameters/axes that led the group to a particular decision have been laid out. things like (but not limited to) "how severe the problem is for certain user groups", "how many user groups are affected", "how easy/hard it is to achieve a better outcome/implement a solution", "are there workarounds that make the problem circumventable".

so clearly the axes have been defined, just that nowhere it has stated, in a simple way, how those axes influenced a decision: when an SC tended to affect more users, was easier to implement, and had no workarounds, it generally ended up in Level A; on the other end of the spectrum, if an SC tended to not affect as many users, or was harder to implement, or there were workaround (albeit laborious/convoluted) that made them less of a showstopper, then an SC tended to be lumped more towards Level AA or AAA. this seems fairly non-controversial statement (though of course not elegant, but far from a "definitive formula" which nobody has actually been asking for), but somehow there's a strong reluctance to acknowledge this for some reason...

love the humpty-dumpty ... I was going to say 'fundamentalism' hence the holy scriptture analogy, but thought that they may be considered inappropriate, and I don't intend to be offensive.

[electronicwoft]

I've read all the comments, but there are so many now that I'm not sure if I've missed something:

• there is a definition for A, AA, AAA
• the definition is not satisfying for some because it is not precise enough.

However, a better definition is not possible because the SC levels are already assigned and even if we were to come up with a new definition, the existing levels would no longer fit. But they can't be changed (so easily). Therefore: Why not live with the fact that there is a definition that may not be perfect, but is still comprehensible. @patrickhlauke: The 3 levels are not just Humpty Dumpty ...

agreed @JAWS-test that it is what it is and we have to live with it, but this does not preclude providing an addendum or additional information rather than formulae.

My central concern is the various 'churches' that have filled the void in lieu of a satisfactory explanation thereby making a nonsense of conformance altogether .... there is nothing preventing an explanation of the original process or directly addressing the ways in which - in my view - these levels are being misused in these growing orthodoxies in a working group note or whatever ...

[JAWS-test]

@patrickhlauke and @electronicwoft: ok, thank you very much for the answers. I can understand that. I.e. the point is that A should defined as essential, no workaround possible, easy to fix, without restrictions on the design

AAA would be defined as: not important, there could be workarounds, difficult to fix, may affect the design or content ...

AA is somewhere in between. It won't be so easy to find a good formulation for this. But I understand that the description of the process for determining the levels is not identical to the definition of the levels. However, in terms of content, the definition will not provide much more than the explanation at https://www.w3.org/WAI/WCAG22/Understanding/conformance#levels

[chaals]

I'd dispute calling AAA "not important" in particular. It's there because it makes a material difference to some people with disabilities.

My very rough mental model, that's now two-and-a-half decades old (going back to formulating the levels for WCAG 1.0), is:

Level A are things that are outright showstoppers - if you don't do them, saying you've created an accessible resource is in the region of a bare-faced lie. Not having a tall narrow set of steps for a wheelchair user to get to the bathroom might be an analogy.

Level AA are the things that can be worked around, but at great effort - some people with disabilities will be unable to effectively use a resource that fails one of these. Removing a couple of low steps that some people could help most wheelchairs over, adding a rough curb-cut, not having a ramp at a 35º angle.

Level AAA are the things that make the difference between "more or less accessible" and "equality of opportunity" - putting the ramp front and centre of the entrance, a light gradient, broad enough to go in company, no corner so tight that many wheelchairs can't actually use it.

When it comes to agreeing exactly where a given requirement fits, given that in different cases it will manifest in different ways and at different levels of severity, means there is no magic formula.

Think about describing images: Sometimes it's crucial, like the map explaining where a shop is or the menu icon. Sometimes it's a picture of the sponsor's CEO, who actually looks like a stock photo, or an actual stock photo - probably a "conventionally pretty" woman because that's what the media still does - added to a newspaper article for no obvious reason.

[cstrobbe]

When we were working on WCAG 2.0, there was a document titled Requirements for WCAG 2.0. I think criteria for conformance levels were brought up in discussions about that document but they were not included in it. As Greg Vanderheiden pointed out in his 3 June comment, the conformance levels assigned to new criteria were based on what we could reach consensus on.

One undocumented rule that we used is that anything that affected visual design would not be at Level A, which is why the most basic contrast requirements (WCAG 2.0's SC 1.4.3 and WCAG 2.1's SC 1.4.11) are at Level AA.

There is also Requirements for WCAG 3.0; based on a brief look, this document does not discuss criteria for assigning conformance levels to success criteria.

[alastc]

I think the main point of the thread is (or should be) to encourage people to apply severity appropriately. I'd suggest a little change to the last suggestion I could find:

Conformance levels (A, AA, or AAA) are not indicative of issue severity for a particular instance of an issue. The severity or impact of any one issue on a given site should be determined by the criticality or importance of the content, as well as how easy the site is to operate and understand.

---

Sidenote: When we report issues the severity is completely separate from WCAG level. I don't know about other agencies, but we've employed people who have worked at other agencies and none were surprised by this approach or asked why we didn't use WCAG levels.

The only time I've come across people asking about whether to use the WCAG level for prioritization is development teams who are new to accessibility. In a scenario where they will have to work through all the A/AA issues anyway, it isn't the end of the world if they do the level A ones first. However, we do encourage them to use the severity and t-shirt-size columns to determine priority (so easy-to-fix blockers are first etc).

[patrickhlauke]

@alastc that'd be a good start, yes. then, if it truly is a case of "SCs have been grouped into levels based on similarity (on the various axes of consideration)" it'd be good to mention that too, to make it even clearer that it's not a hierarchy of importance, but just a very rough set of buckets (which still doesn't explain why this was done - if #allSCsMatter - what each bucket actually represents, but it'd be better than nothing/conjecture)

[patrickhlauke]

perhaps relevant, as the missing piece for me at least has been "yes, we understand that SCs were placed in categories/levels because the WG decided they should go there, but surely the WG must have had a working understanding or definition of what each category/level actually represents in order to make that judgement call" ... this seems a good starting point https://lists.w3.org/Archives/Public/w3c-wai-gl/2002AprJun/0266.html

[GreggVan]

Let me start by first asking -- why do people want to know? Usually it is:

1. "so I know which ones are most important to do --
2. or to do first"

The answer to the first (which are more/most important) is

• they are all important - or the time to put them in and create the conformance document and the Understanding WCAG 2.0 and the techniques documents (which all took an ENORMOUS amount of time to do) would not have been spent by people who were all volunteering and had jobs and families as well.

The answer to the second (which should I do first) is

• that is difficult to answer quickly - and different people have put forward different strategies and all are good. here are some

  • well first there are always low hanging fruit. Things that you could do quickly that you could get a whole bunch done. Just do them rather than having them all stand in line behind some one item that is really hard and takes a long time to fix. That doesn't mean you should just do them in order of ease. See the next point about importance. But don't let one single important item block you from doing 30 other easier ones that are important for someone.
  • do them in order of importance if you can figure out what that is. Doing a lot of little things that don't really matter much and only make the page marginally different is not as important as doing the important ones that really create real problems.
  • there are no little ones. They are little for one individual, but they may be showstoppers for other people. This particularly happens in the area of cognitive language and learning disabilities where people think that these are just "usability" issues when they are in fact issues that prevent people in these groups from using things.
  • do the showstoppers first. Those things that basically cause a page to be completely unusable. A couple examples are pages at flash and can put someone into a seizure before they even know there is a problem on the page and pages that auto play audio (that does not stop after a few seconds) so that an individual using a screen reader suddenly cannot hear their screen reader. But also see the comment above about the fact that everything can be a showstopper for somebody so don't have to narrow a definition of showstopper.
  • etc

OK that now raises the question:\ Why are there three levels if it isnt importance. What do the levels mean?

That is a great question - that is somewhat lost to time. Here is the answer in short - followed by more detail.

• all of the provisions were important (including AAA) so the levels were not created to indicate importance.
  • items are in AAA only because we could not require them for one reason or another such as
    1. they could not be applied to all pages and we could not define which pages they should apply to. So they could not be required (for all pages - which is what WCAG does)
       2. we could not find any way to objectively test whether they were done or not (so could not do them)
       3. It would be impossible to do them on every page (e.g. requiring sign language translation of every page - since there were not enough people who could do the translations over the course of a year for all the pages that were generated in a minute on the web. )
       4. etc. [Some of them are captured here](https://www.w3.org/WAI/WCAG22/Understanding/conformance#levels
       5. )
          • so items were in AAA not because they were less important but for some other reason where the working group did not feel they could be required for all pages.
          • level AAA was created to keep things into the guidelines that could not be applied everywhere -- but should be done whenever possible.

OK that explains level AAA (sort of). So why are there two more levels (A and AA) instead of just one more (other than AAA)?

That is the one that is most misunderstood I think.

• And all sorts of rationale are provided mostly in retrospect.

• @patrickhlauke found notes from back when we were working on it (Thank you Patrick for finding that).

  • This gives you and idea of what the 2.0 working group was trying to do. But it turns out that when we tried to apply these rules to sort the provisions -- we kept finding exceptions.
  • If you look at these and the final sort of WCAG 2.0 you will see they don't quite fit. Some fit but others don't.
  • There is also different interpretation by working group members as to what the words in each of these "sorting rules" meant.
  • So this document is very useful in getting insight into what the working group members were thinking about - but much of this was done AFTER many/most SC were sorted and we were trying to figure out what we used as rules for doing it to shorten discussions for each new one.

• And most of the writeups done since -- are rationalizations for how it was done and not a list of the rules that were used in actually doing it. At least not rules that were strict sorting rules. They were more guidelines and starting points for where to put things.

So that leaves -- Why were there 3 levels instead of 2?

The bottom line is that -- we could not get any consensus when we tried to have just two levels

• Required
• Recommended

At one point - there was a concerted push to have WCAG look more like other standards - where there were only two levels: required and recommended (equivalent to SHALLs and SHOULDs.) There was even a majority of the group that wanted to have two levels.

The problem was that half of the group that wanted two levels wanted to create it by collapsing everything in AA into AAA, and the other half wanted to collapse AA into A.

So in the end - the one thing that everyone could agree on was the three levels.

And the items in the three levels were determined in the same (consensus) manner. Each item was placed in the location where the working group finally put them.-- often after long discussions on multiple meetings -- and always using feedback from the many public-releases-and-requests-for-comments that we received.

All the lists of ideas for how to sort them from the past and the lists that have been made retrospectively to explain them are all good indications of the factors that went into deciding which went where -- but they are not absolute determinants -- and you will find things that seem to not quite fit or not fit that set of criteria. That is because they were factors that were considered - not criteria.

If you are looking for the absolute accurate description of why each item ended up in the level it did -- here it is.

The SC ended up at the level that the working group could reach a consensus on putting them after sometimes long discussions and multiple releases for public comment.

That is the only accurate and detailed answer.

• all the other retrospective lists of reasons (including the one from the original working group) are good lists of factors that were considered in the process. But not deterministic items.

Looking back is only so helpful. And mostly just for the similar task that we will have for WCAG 3.0

As to using levels for the two questions at the top (i.e.,. "Which are more important ?" and "Where should I start?") the answer is DON'T.

• They are bad indications of both.
• There are other much better strategies and better writeups and documentation of both.

But NEVER take levels as levels of importance to people with disabilities. They were NEVER that.

Hope this is helpful.

Best

Gregg
Co-Chair WCAG 2.0 working group

[patrickhlauke]

[edit: actually, I give up, this is just going further in circles and there's no real point here anymore]

[electronicwoft]

I think the main point of the thread is (or should be) to encourage people to apply severity appropriately. I'd suggest a little change to the last suggestion I could find:

Conformance levels (A, AA, or AAA) are not indicative of issue severity for a particular instance of an issue. The severity or impact of any one issue on a given site should be determined by the criticality or importance of the content, as well as how easy the site is to operate and understand.

Sidenote: When we report issues the severity is completely separate from WCAG level. I don't know about other agencies, but we've employed people who have worked at other agencies and none were surprised by this approach or asked why we didn't use WCAG levels.

The only time I've come across people asking about whether to use the WCAG level for prioritization is development teams who are new to accessibility. In a scenario where they will have to work through all the A/AA issues anyway, it isn't the end of the world if they do the level A ones first. However, we do encourage them to use the severity and t-shirt-size columns to determine priority (so easy-to-fix blockers are first etc).

this is precisely what is wrong with the current situation in my view.

if severity is determined by criticality then it is tautological and superfluous.

As I have said before, as a training aid for developers it is at best vague and at worst misleading because it is a value plucked out of the sky by testers who - in my experience - wouldn't know their ass from a hole in the ground when assessing user impact ...

So let's abandon severity once and for all and look at priority.

"the importance of the content as well as how easy the site is to operate and understand" is neither readily testable nor especially practicable given that it is likely only confirmed by extensive usability testing which is a nice to have most organisations place behind accessibility in their list of priorities.

What's more, 'the importance of the content' is, if you break it down' just another way of saying 'the chances of this or that preventing a user from completing a given user task is 100%' which is basically saying it's a Sev1 issue.

And, again, not something I'd leave up to the average tester.

So let's put aside priority as defined above also and concentrate on why a tester needs to assign a value to a defect in the first place or at all.

The reason is so that whomever is resolving said defects then has what is effectively a chronology to help structure their workload.

As soon as levels of conformance - whatever these might mean or how they are defined or by what process they were conceived - is removed from the calculation of this chronology, then the ultimate organisational goal of achieving a given level of conformance for a defined scope of web content is jeopardised.

tester X says that a single failure of 1.1.1 is a S4 and hey presto it isn't resolved before production and, given the awkward nature of the WCAG 2.x conformance model, may mean that conformance requirements 1, 2, and 3 are not satisfied and any claim of conformance for that scope or part of it is no longer valid.

It simply doesn't matter what each level of conformance means or how a given success criterion was assign to it - as soon as levels of conformance are obliviated by arbitrary assignments of severity or priority, or importance, or untested perceptions about ease-of-use, then the specifcation becomes valueless.

This is why there must be another factor or factors that can be assigned by testers objectively such as frequency or scope or some other categorisation that don't rely on a moistened finger in the air or experience or sixth sense or divine guidance to formulate a chronology based on levels of conformance.

Any proposed addendum must make it clear that traditional conceptions of severity-as-user-impact or criticality as just another name for severity or any other similar utterley subjective or untraceable or untestable factor is inappropriate for setting a chronology.

[electronicwoft]

[edit: actually, I give up, this is just going further in circles and there's no real point here anymore]

that's sad to hear, Patrick - there is a point and it's to make WCAG 2.x more usable for the people who are stuck with it.

[JAWS-test]

if all levels are important and, above all, if there is no difference between A and AA, then WCAG 5.2.1 makes no sense. There, a precise distinction is made between

• A (“minimum level”)
• A+AA
• A+AA+AAA

If they are all equally important, WCAG could also offer that someone achieves conformance with

• AA
• AAA
• A+AAA
• AA+AAA

[electronicwoft]

the key there is ''equally' ... some are more equal than others! No-one has said that all success criteria are EQUALLY important, only that they are important ... which is about as useful as saying that they all are success criteria.

this is part of the problem with this thread - terms such as severity, priority, criticality, importance, etc. are undefined ...