No definition of what constitutes an A, AA, or AAA conformance Level

[Helixopp]

There does not appear to be any clear definition of what parameters a success criteria must meet in order to qualify for any particular conformance level.

In order to be considered Conformance Level A does a success criteria have meet a certain level of severity, as in if not met it will severely adversely impact the user.

Likewise is an AA Conformance Level less severe/impactful? And so on.

Or is it that Level A Conformance is attributed to easy fixes, slightly more complicated issues are deemed AA Level, etc.?

Accessibility consulting companies are assigning severity levels to WCAG Success Criteria (critical, moderate, serious, etc.)which seem to be largely based on conformance level.

This is extremely subjective, and dangerously misleading. Lack of transparency of what criteria are required to be met for each conformance level is partly to blame.

If such information does exist it is too difficult to find.

[AutoSponge]

Paraphrased from here:

• A = "minimum level of conformance"
• AA = recommended as general policy
• AAA = not recommended as general policy (too difficult)

[Helixopp]

Please read the thread. What qualifies as A level AA level and AAA level is not defined.

[mbgower]

IMO, this issue is unlikely to ever be resolved because of its contents.

The issue text makes a statement:

1. that the rationale for what was put at different levels isn't defined.

Then it asks a few questions:

2. can severity can be inferred from level?

3. can ease/complication in remediation be inferred from level?

Then it makes an unsubstantiated assertion:

4. "Accessibility consulting companies are assigning severity levels to WCAG Success Criteria...largely based on conformance level.

Then it makes a conclusion:

5. "Lack of transparency of what criteria are required to be met for each conformance level is partly to blame."

For No.1, @alastc already pointed to existing details in the Understanding document. Discussion has noted that that differs from what was in 1.0, that there is not enough granular detail to map each SC to a level, and that there was no specific formula used in any case. Others have noted that since pretty much every jurisdiction just says 'do A and AA', then the outcome ('Meet these 4 dozen') is a lot more important than an explanation of how criteria were assigned to A or AA.

For No.2 and No.3, the answer is "no". I have not seen anyone suggest the answer is "yes". Some have noted a possible case for priority.

For No.4, I continue to wait for evidence it is the case. It's already been noted by a number of responders that if you use the word "priority" instead of "severity", level AAA SCs are clearly deprioritized, since no jurisdictions require them. I am unaware of any organization that would use the A/AA level of an SC as the main basis for severity ratings, hence my request for examples.

For No.5, I find the statement confounding. First, the success criteria are listed within a conformance level. If you only want to meet level A, you do the SCs labelled "A". Second, each of the SCs, in turn, lists in its normative language what is required to meet that specific SC. Third, this is all built on the supposition that people out there are using level as the primary driver of severity AND an implication that if there was a published list of how the SCs were assigned to levels, this would make a material difference to how these teams operated.

If someone has a suggested way of satisfying the issue, I invite them to create a PR, otherwise, I'm inclined to mark this as a Discussion.

[patrickhlauke]

For No.1, @alastc already pointed to existing details in the Understanding document. Discussion has noted that that differs from what was in 1.0, that there is not enough granular detail to map each SC to a level, and that there was no specific formula used in any case. Others have noted that since pretty much every jurisdiction just says 'do A and AA', then the outcome ('Meet these 4 dozen') is a lot more important than an explanation of how criteria were assigned to A or AA.

and others have noted that what is currently in the understanding document does not sufficiently explain things: the document explains what factors were considered, but not what the levels ultimately mean. then, there's been a circular "if an SC is at Level X, it means the WG reached consensus that it should be at Level X" without explanation, still, what Level X actually represents.

[mbgower]

it means the WG reached consensus that it should be at Level X" without explanation, still, what Level X actually represents.

I invite you to take a stab at writing such language, and on where you would position it in the non-normative document. My suspicion is that it will be very difficult to get consensus on what level A means and what level AA means, especially because you're going to be forced to reverse engineer this and show that based on what you write each of the SCs fits neatly into that category. I tend to side with Jonathan that it is arbitrary and not really an issue, since A/AA are glommed into a single conformance goal.

I'm going to bow out of this discussion, but will be happy to be proven wrong in my pessimism.

[bruce-usab]

What qualifies as A level AA level and AAA level is not defined.

The factors are explained, and the consensus assignment is more of heuristic than an algorithm.

This old WCAG_2.x Priority levels discussion may be of interest. The included tabular analysis was not formally endorsed by AG WG (but that is true for most of the wiki).

@Helixopp please be encouraged to propose edits to Understanding which you would read as addressing your concern.

[alastc]

I'm afraid I share @mbgower's pessimism that there is a solution to this, and the reason is simple:

It is too complex to boil down to some bullet points.

Gregg outlined the process, and that was similar in 2.1 and 2.2. Is it arbitrary? That implies it is chosen by a person, but it is not chosen by any one person, it is a multi-stakeholder group consensus process.

If you are skeptical about the complexity, have a look at the page Bruce linked to - The old WCAG_2.x Priority levels discussion

The Essential / Easy / Invisible / All Content factors are like weights and balances, some positive some negative. They are also not binary (despite the yes/no in the data cells), each is on a continuum which factors into the decision.

What that means is that, when looking at one SC you can't see it as higher priority than another. Also, who would that be a higher priority for? The user? The developer? Feasibility and User-need are like a cost:benefit ratio. If it's easy and obviously important, it goes up, if it's hard and the user has workarounds, it goes down. And that's only 2 of the factors.

Including the discussions and decisions about that on a per-SC basis would probably fill up more pages than the understanding documents (not that anyone suggested doing that, I just mean that it's a lot of discussion).

We could provide a general description of what the factors were and the consensus process, but I don't think that answers the original question, because there isn't a simple answer.

Another thing we could add is a statement along the lines of: Prioritization of accessibility issues should include the context, for example, alt-text on a 'submit' button would be a more severe issue that alt-text on a logo in the footer of a page.

However, I tend to agree with Detlev, it's done pretty well for 16 years so this doesn't seem urgent.

---

Secondarily, I don't think these comments should stand without rebuttal:

It’s not defense say well we all agreed on it. If someone where to bring to the attention of EU parliament...

It is a stakeholder process, not a scientific one. Producing something that satisfies the various stakeholders can't be done by a pre-set formula. I think places like the EU parliament are looking for this kind of process! It means we have done the work of balancing user and industry needs which makes implementing regulations easier.

The document itself calls Level A a minimum level of accessibility

Yes, with three levels, that is the one you meet without meeting the others, i.e. least number of requirements met, therefore minimum. It isn't speaking to severity.

I know many WCAG 2.0’ers that were in that room. Some did not have disabilities. Some did not have Assistive Technology background, some did not have programing background.

No one has everything, and some people did have disability, an AT background, or a programming background. But more to the point, after the initial group discussions it goes to a wider audience who all get to comment. That's where you get the widest breadth.

I find it very strange that we create a standard that includes levels of conformance that we don’t require or recommend. A standard is a set of requirements.

They provide a route for people who want to do more. Not many people do, but we have had clients who have requested testing and implementation of AAA level criteria. The discussions in WCAG 3 are around making a more effective and encouraging route for people to do more.

If you have an eye on the regulatory level and think "should a person or organisation get sued for not doing this?". It is pretty common to think "yes", for some organsations, e.g. national / goverment type orgs. But then you think about small businesses, schools, and other smaller places and the answer becomes "no". Therefore including things we think should be done, but can't require of everyone, becomes an outlet for those requirements.

[Helixopp]

Arbitrary does not imply one person acted alone. “If you describe an action, rule, or decision as arbitrary, you think that it is not based on any principle, plan, or system. It often seems unfair because of this.” – pulled from a dictionary

I don’t how you can argue that it has done well for 16 years when only a couple of years later, a WCAG 3 was requested because 2.0 wasn’t sufficient, and confused people. Furthermore, it has been inconsistently applied in U.S. litigation, and this Working Group is well aware of the numerous complaints about its opaqueness and complexity.

It has only “done well” because there isn’t any alternative to date.

This response is akin to saying “we got away with it, so what?”

That is bad for the integrity of the WAI and the W3C.

Less than 3% of websites are accessible according to the most recent WebAIM study. Is that indicative of WCAG’s success? Of course, it indicates an unsuccessful attempt to provide universal guidance that can be widely interpreted, implemented, and sustained. We should be honest with the shortcomings and put in the effort to address them.

As I stated on June 4:

My ultimate point for resolution is this:

We need to publish a warning/caution/note that explains what conformance levels are not (they are not levels of severity or priority). It should be made clear that there isn’t any particular standardized method that was used to define or assign them.

APA suggests we publish it as an errata

[alastc]

Hi David,

If you're going to quote a dictionary, start with the definition: "based on random choice or personal whim", which is not the case at all, as demonstrated above.

I don’t how you can argue that it has done well for 16 years when only a couple of years later, a WCAG 3 was requested because 2.0 wasn’t sufficient, and confused people

It's done well because, despite other options, it has take up around the world. Nothing is perfect, and people are always looking at the next new, hopefully better thing. Other people have tried, but it's a hard thing to do, and the other attempts were missing the wide stakeholder input and open process the W3C provides.

You seem to think the consensus approach is a downside to defining this sort of thing, but it is that approach that leads to stakeholder buy-in.

I'm not sure what point you are trying to make referencing the WebAim survey. WebAIM cites WCAG as how it measures accessibility, so when sites fail to measure up to that standard, is it a failure of the standard, or a failure of take-up, or a failure of the sites? It’s a bit like saying: "Hey, when I measure all these items, which are all supposed to be 30cm, with a 30cm ruler, a lot of the items are different lengths! What a rubbish ruler!”

One thing I'm pretty sure of, is that explaining why SCs go into A or AA is not going to solve that problem. There are lots of things to improve, but I think understandabillity, clarity (of the requirements and testing), and incorporating more requirements (particularly COGA ones) are more important things to work on.

If APA has an official comment on this topic, please link to it.

[Helixopp]

The focal point on the process of how conformance levels were decided does not effect my requested resolution. The problem I brought to light is that without a disclaimer/warning/caution on what conformance levels don’t/do mean they are easily misconstrued and are being misapplied. This is and has been occurring.

I will request, again, that we focus on the issue, as I have stated above.

The perceived success of WCAG is also immaterial. It is clear that conformance levels are not a statement of Success Criteria priority or severity. If they can be misconstrued as such, and I have pointed out that they are, then we have a duty to remedy that. You are welcome to vote against that remedy but I have made my proposed resolution clear. Do we need to put this on an AGWG Agenda? I discussed it during the new business portion of APA’s last teleconference. Since APA is responsible for formal horizontal review I will discuss with the chairs on putting it in their Agenda so we can make a formal response.

By the way, Your definition provided for arbitrary does not change my point. Votes can be cast based on personal whim when the subject matter does not provide clear guidance.

To put this point into perspective we can use the demotion of Pluto’s demotion to Dwarf Planet as an example

In August 2006, the International Astronomical Union, or IAU, wrote a new definition of “planet” that left Pluto out. The new definition required that a body 1) orbit the sun, 2) have enough mass to be spherical (or close) and 3) have cleared the neighborhood around its orbit of other bodies. Objects that meet the first two criteria but not the third, like Pluto, were designated “dwarf planets.”

In this example 3 criteria were specified to meet the definition and the scientific community voted on whether or not the subject met that definition.

What happened in Conformance Levels for WCAG was the Working Group said that they considered numerous criteria but did not specify what specific criteria needed to be met to qualify for each conformance level. Without specifying the requirements for each conformance level the vote is arbitrary.

In order to not be arbitrary the criteria for conformance Level A AA and AAA need to be stated. Minimum level of conformance is a vague general statement for Level A. It does not specify what specific considerations determine that.

This is actually a perfect example because the situation is similar to ours in that they realize they needed criteria for a scientific definition that was previously understood.

https://www.sciencenews.org/article/pluto-planet-vote-status-definition-demotion#:~:text=The%20new%20definition%20required%20that%20a%20body%201%29,the%20third%2C%20like%20Pluto%2C%20were%20designated%20“dwarf%20planets.”

[alastc]

Hi David,

If your focus is to prevent the levels being mis-applied, what about the suggestion I made above?

Another thing we could add is a statement along the lines of: Prioritization of accessibility issues should include the context, for example, alt-text on a 'submit' button would be a more severe issue that alt-text on a logo in the footer of a page.

If that's the main issue, then the process aspect is off-topic.

[yatil]

This is a reminder that WCAG is a political consensus document, not a technical specification. So the combination of factors that resulted in an SC’s level are different and individual for the circumstances and impact of each success criterion, while also ensuring that WCAG covers a broad variety of disability access at any level. There is no requirement in the W3C process that every part of a published document must follow a strict rationale, and in fact, that would probably make it impossible to create real-world standards.

The group agreed that the SCs that are level A are enough for A conformance, A+AA are enough for AA conformance. They were never meant as prioritization, nor has the WG or WAI ever claimed they were. I agree there are misunderstandings; maybe a simple sentence “Levels do not necessary correspond to prioritization.” is sufficient to address the issue.

[patrickhlauke]

The group agreed that the SCs that are level A are enough for A conformance, A+AA are enough for AA conformance.

purely from my side, i just still fundamentally dislike this tautological explanation. when i get a moment, I want to at least resolve this bit.

[yatil]

The group agreed that the SCs that are level A are enough for A conformance, A+AA are enough for AA conformance.

purely from my side, i just still fundamentally dislike this tautological explanation. when i get a moment, I want to at least resolve this bit.

I didn’t say that I like it 😂 It’s just what happened 😬

[kiara-stewart]

I agree there are misunderstandings; maybe a simple sentence “Levels do not necessary correspond to prioritization.” is sufficient to address the issue.

What about these options -

1. Issue severity and prioritization should not be solely determined by WCAG level.
2. WCAG level should not be the sole factor in determining issue severity or prioritization.

[mbgower]

@kiara-stewart I think we can even make it more explicit than that. I'm going to suggest the following be added after the bulleted list in the Understanding Levels of Conformance section :

Conformance levels do not correlate with issue severity, and the conformance level of a failing success criterion (whether A, AA, or AAA) should not be used to apply severity to individual issues. Instead, the impact of any one issue will be dependent on the context of the page and the real-world impact to users.

[GreggVan]

Correlate is a technical term and has a specific meaning - and in fact, they do correlate. But they don’t define it.

Hmmm I was thinking something like...

There are a lot of factors that went into conformance levels. None of them were “this is what is most important or critical for users of this particulat site”. A better measure for severity or priority might be the impact that an issue or failure would have on the accessibility of the particular content or site you are working on.

But that would require that they understand accessibility enough to judge the impact. And most don’t.

Rather than saying that conformance levels are NOT the best measure - I think it would be better to instead (or also) say what WOULD be the best way. But do it in a way that does not require that they understand accessibility or the impact of different SC. Do we have a “note” or other document somewhere that provides practical advice for prioritizing? This would be the most useful.

[mbgower]

Rewriting without using the word "correlate": [updated as per @GreggVan suggestion]

Conformance levels are not indicative of issue severity for a given site. The conformance level of a failing success criterion (whether A, AA, or AAA) should not be used to apply severity to individual issues. Instead, the impact of any one issue will be dependent on the context of the page and the real-world impact to users.

[GreggVan]

looks good. Nice job. I would perhaps just add “for a given site” at the end of the first sentence so it reads.

Conformance levels are not indicative of issue severity for any given site. The conformance level of a failing success criterion (whether A, AA, or AAA) should not be used to apply severity to individual issues. Instead, the impact of any one issue will be dependent on the context of the page and the real-world impact to users.

[DuffJohnson]

FWIW, I've tended to regard various Level A SC as predicates for Level AA, and so on. For example, 1.3.1 seems like a predicate for many other SC (including at Level A).

Perhaps a "predicate tree" establishing which SC are logically required to fulfill other SC could help in usefully (and objectively) organizing SC in a manner that would inform on priorities?

[kiara-stewart]

Conformance levels are not indicative of issue severity. The conformance level of a failing success criterion (whether A, AA, or AAA) should not be used to apply severity to individual issues. Instead, the impact of any one issue will be dependent on the context of the page and the real-world impact to users.

@mbgower I like your latest version! Maybe we should consider using Harvard's accessibility severity matrix to clarify and simplify the language even further -

Image 1 Description: A 2 by 2 matrix that identifies the severity of accessibility issues as either Low, Medium or High. The vertical axis of the matrix corresponds to the range of operability and perceivability of content, with operable or perceivable content at the bottom of the axis and inoperable or perceivable content at the top. The horizontal axis corresponds to the range of relevance of content with less relevant content to the left of the axis and very relevant content to the right.

[kiara-stewart]

Frankenstein copy: Conformance levels are not indicative of issue severity. The conformance level of a failing success criterion (whether A, AA, or AAA) should not be used to determine the severity of individual issues. Instead, the impact of any one issue on a given site should be determined by the criticality or importance of the content, as well as how easy the site is to operate and understand.

[mbgower]

@kiara-stewart Thanks for the suggestions. You have worked the notion of site, suggested by @GreggVan, later in your rewording and that seems to work well there. If other participants such as @Helixopp can provide thumbs up to Kiara's suggestions, I'll get a PR done for this (unless you want the honours, Kiara).

[Helixopp]

Thanks, Mike. I don’t think we need to rush to PR, although I appreciate the urgency. I would like to give APA a chance to review and respond. It will take a couple weeks, due to already scheduled agendas.

[mraccess77]

Saying they are "not indicative" - rather than saying "on a whole they may not be indicative" could cause confusion of why there are even levels A and AA at all - so I think it only creates more questions. I'd rather adjust the language a bit rather than saying the level has no bearing at all.

[bruce-usab]

I would like to give APA a chance to review and respond. It will take a couple weeks, due to already scheduled agendas.

@Helixopp would you share your inquiry as put to APA? I must confess only passing familiarity with the APA Style Guide. Is it structured such that one can cite chapter and verse as to the convention WCAG 2 is failing to adhere to? (And, if so, please share that too.) Thanks!

[Helixopp]

Here’s a link to the charter: https://www.w3.org/2023/07/apa-wg-charter

Basic responsibility is horizontal review of all W3C specs, evaluating their impact on accessibility and responding with recommendations, revisions, etc. My inquiry was to respond as a working group with recommended language in the proper format ( it has been informally suggested in one of our working group discussions that this be an errata) Sent from my iPhone

[kiara-stewart]

I feel like we're nearing a resolution! To wrap up my day, I'll offer one final content suggestion. I've merged the first two lines and removed the word "indicative"(cc: @mraccess77). I believe this revised copy clearly conveys that they should not completely abandon conformance levels, nor should they rely solely on them for prioritization or to determine issue severity.

Conformance levels alone should not be used to determine the severity of accessibility issues. Instead, the severity of accessibility issues should be assessed based on the criticality of the content being reviewed, as well as the operability and perceivability of the given site.

[mbgower]

Thanks, Mike. I don’t think we need to rush to PR, although I appreciate the urgency. I would like to give APA a chance to review and respond. It will take a couple weeks, due to already scheduled agendas.

All that a PR does is give the WCAG 2 task force something to consider and act on. This issue was opened against the WCAG github repo, and that is our process. Until such time as this is even put before the working group, I'm going to suggest involving another body seems premature.

@bruce-usab

would you share your inquiry as put to APA? I

I believe @Helixopp is referring to Accessible Platform Architectures (APA) Working Group I was equally confused by the unlinked three-letter acronym, but that is my assumption.

@Helixopp, from their page, they have a process

To request review of a GitHub issue from the APA, add the “a11y-tracker” label to the issue. If the APA files an issue or agrees to comment on an issue, the label will be changed to “a11y-needs-resolution”. To request review outside of a GitHub issue context, send email to group-apa-chairs@w3.org.

I am treating your comments as a request for that. If I am misunderstanding, feel free to remove the label.

[mbgower]

Say they are not indicative - rather than saying may not be indicative seems to cause confusion of why there are even levels A and AA at all - so I think it only creates more questions. I'd rather often the language a bit rather just saying they have not bearing at all.

@mraccess77 There are a few typos in your comment making it a bit hard to understand, but are you saying that there is some level of issue severity associated with a level? I would like to understand what you believe that is. I have not seen any other comments in the thread stating a relationship.

As I understand the purpose of this issue -- or at least some of the various reasons that have been given over its long comments history -- the desire is to make a clear statement that Conformance levels should not be used to determine severity. So turning around and implying that the levels do affect severity to some degree seems very problematic to me.

At a quick glance, I see absolutely no commonality between all the SCs at Level A and a pre-determined level of severity from the resulting failures. Certainly there are a few SCs where a failure is likely to be treated as a high sev. We have been talking about those in recent WCAG 3 discussions. But even there, the context of the page will I suspect strongly affect how a team would actually classify it. But there are other level A criteria where almost the complete opposite is true; where it would be an unusual situation indeed if a failure of the SC resulted in a high sev issue.

I will note that I considered mentioning something about priority in my proposed text because several commenters have suggested ways the levels could be used to inform priority. However, I suspect that is going to take a lot of language crafting. I just wanted to get an easy win in regard to the original issue, and until your comment, I thought 'lack of correlation with severity' was the thing everyone was aligning on.

[mraccess77]

I updated my comment to make it more clear what I was trying to say. We have several SC with minimums and enhanced - e.g. contrast. While there are many factors that are in play including visual impact on the page, ability to make the changes (easy), etc. it is pretty clear that not meeting minimum contrast ratio has a higher severity impact to users with low vision than meeting a minimum but not an enhanced level.

One of the columns in the original table was "essential" https://www.w3.org/WAI/GL/wiki/WCAG_2.x_Priority_levels_discussion and while not all Level A items are essential (most were) - most of the original WCAG 2.0 AA items were not essential - with 3 exceptions - likely due to complexity of implementation. So, on a whole A items tended toward essential while AA tended toward non-essential with some exceptions.

So, while the conformance alone does not create a priority in some cases the conformance level have some relation to severity. Furthermore, all of the original AAA except for 1 were listed as not easy - which could indicate a lower priority. Saying that all issues are always contextual creates some challenges as well as today that would always mean severity and priority must be determined manually.

[mbgower]

@DuffJohnson

FWIW, I've tended to regard various Level A SC as predicates for Level AA, and so on. For example, 1.3.1 seems like a predicate for many other SC (including at Level A). Perhaps a "predicate tree" establishing which SC are logically required to fulfill other SC could help in usefully (and objectively) organizing SC in a manner that would inform on priorities?

To make the obvious point, you are talking about priority, not severity, right? The stated issue is about severity.

Your comment has some resemblance to Jon's recent comment (although he mixes in severity as well). As Jon points out, if there is a "Minimum" and "Enhanced" version of the same basic requirement, then it strongly suggests there's a predicate. I'll state the obvious and note that something that fails at its minimum requirement version generally also fails its enhanced requirement version.

Here's the thing about priority, though. Beyond the minimum/enhanced pairs, a lot of things about priority are inferred, and assume we attempt to create lanugage to address priority and not severity, I think we are going to see a whole bunch of different suggestions on how priority could be assigned based on the level or nature of the requirements.

Does anyone really think we can arrive at consensus on anything about priority but vague comments? I've taken a crack at tackling what I thought was the relatively simple task of stating that one cannot infer severity based on conformance levels. I feel like even that lighter task is doomed to a lot of iterations.

BTW, I still have yet to see anyone produce evidence showing detrimental use of improper interpretations of levels in regard to severity, which is the stated problem in this issue. (Priority is not even mentioned.)

[GreggVan]

I am not sure of the desired outcome or objective of this thread. Several observations.

1. Severity - depends on WHO (What is severe to one user is not relevant to another. and things that seem to be "usability" to some users are showstoppers for others. There is no absolute scale of severity that people will agree on unless you want to pick which disabilities are more important. And cognitive always seems to fall into "usability" rather than showstopper even though they are showstoppers for some users )
2. Priority - is context dependent (What is the highest priority on one site may not be on another. And there are many factors that go into setting priorities)
3. Lots of discussion, but it would be useful for each posting to propose exactly what language one suggests using where. Discussions in the abstract always seem reasonable even when they contradict or conflict with each other. (e.g. the guidelines should be shorter and more complete, easier to read but objective and technology agnostic, require both high and low contrast, etc) Coming up with specific language to change or add that does not create its own problems is the challenge.

Best G

[Ryladog]

Concerning severity and prioritization I focus and have guided others to rely on the 4 SC in the "Non-Interference" Conformance Section under + whatever is relevant to the content itself (Examples: is it banking app? then:

1. 3.3.4Error Prevention (Legal, Financial, Data) https://www.w3.org/TR/WCAG22/#error-prevention-legal-financial-data

   or, is it a web app optimized for voice control in a car? then XX...should be viewed as Highest/Vital), and then all the rest of the SCs, again relevant to the content.

Conformance Requirements https://www.w3.org/TR/WCAG22/#conformance-reqs

1. 5.2.1Conformance Level https://www.w3.org/TR/WCAG22/#cc1
2. 5.2.2Full pages https://www.w3.org/TR/WCAG22/#cc2
3. 5.2.3Complete processes https://www.w3.org/TR/WCAG22/#cc3
4. 5.2.4Only Accessibility-Supported Ways of Using Technologies https://www.w3.org/TR/WCAG22/#cc4
5. 5.2.5Non-Interference https://www.w3.org/TR/WCAG22/#cc5
6. • 1.4.2 - Audio Control,
   • 2.1.2 - No Keyboard Trap,
   • 2.3.1 - Three Flashes or Below Threshold, and
   • 2.2.2 - Pause, Stop, Hide.

   • https://www.w3.org/TR/WCAG22/#cc5

LEVELS: My recollection of the original thinking/rational (at that time in technology early 2000's) of the 3 levels, was Level A SCs would impact the greatest number AND broadest range of users with disabilities, Level AA SCs were more specific range and tasks to address specific disability types, Level AAA SCs were typically considered going the extra mile for specific disabilities that were not then well covered (or covered at all by AT/Browsers) by technologies that were available at the time.

So perhaps we should consider upgrading that area of the Conformance Section 5.2, in the spec, as a home for Levels definition.

[DuffJohnson]

@mbgower

To make the obvious point, you are talking about priority, not severity, right? The stated issue is about severity.

Priority and severity may be entangled. There may be a useful disambiguation that has some effect on considering representations of "severity". Perhaps it's usefully represented in matrix rather than linear terms, particularly if it's the case that satisfying some SC is logically or practically prior to satisfying other SC. Systemic issues would (I'd assume?) tend to be higher priority than point-issues, even if the point-issue's severity is high, and so on.

Your comment has some resemblance to Jon's recent comment (although he mixes in severity as well).

Agreed.

Here's the thing about priority, though. Beyond the minimum/enhanced pairs, a lot of things about priority are inferred

The technical implications of SC 1.3.1 feel like a "early" predicate for those that would enable solutions for other SC, in the sense that failures in terms of 1.3.1 make other failures more likely and more severe.

Perhaps there's a tree here, in addition to "minimum / enhanced" pairs.

, and assume we attempt to create lanugage to address priority and not severity, I think we are going to see a whole bunch of different suggestions on how priority could be assigned based on the level or nature of the requirements.

Yes, this is a question about framing. I'm not sure that the answer isn't to recognize BOTH priority and severity.

Does anyone really think we can arrive at consensus on anything about priority but vague comments? I've taken a crack at tackling what I thought was the relatively simple task of stating that one cannot infer severity based on conformance levels. I feel like even that lighter task is doomed to a lot of iterations.

Fair enough; I'm a newcomer to these discussions, that's for sure!

BTW, I still have yet to see anyone produce evidence showing detrimental use of improper interpretations of levels in regard to severity, which is the stated problem in this issue. (Priority is not even mentioned.)

My observation as well...

[kiara-stewart]

Whew! Lots of context here! Getting back up to speed after the weekend! I'll provide a brief recap for those who, like me, might need it.

Ask:

Clarify the parameters that were used to determine conformance levels.

Concerns:

• Misinterpretation: Some consulting agencies rank accessibility issues based solely on WCAG conformance levels (e.g., Level A = high, Level AA = medium, Level AAA = low), which misrepresents their real-world impact.
• Defendability: The lack of transparent criteria undermines the credibility and effectiveness of WCAG.
• Future Guidance: There is no standardized method for assigning conformance levels to new success criteria.

References:

• [Unknown] Business Name(s): The one or more companies doing this is unknown, no business names were provided. Other WCAG members in this thread have pointed out that they base severity based on real-world impact regardless of conformance level.
• Understanding Levels of Conformance: https://www.w3.org/WAI/WCAG22/Understanding/conformance#levels
• Priority Levels Discussion: https://www.w3.org/WAI/GL/wiki/WCAG_2.x_Priority_levels_discussion

Main Argument:

• Consensus-Driven Decision Making:
  • Challenge: Relies on qualitative judgments, which can be subjective and not easily replicable.
  • Defense: Reflects real-world complexities. Disability and, subsequently, accessibility exist on a spectrum. Additionally, it mirrors the process of how most government policies are created globally; they are not solely based on data.

Consensus:

• Short term: We’ll update the non-normative documentation to clarify the relationship between severity, priority, and conformance levels.
• Long-term: We recognize that creating a binary framework for leveling retroactively would be extremely challenging. WCAG 3.x? Or, it may be so complex that we revert back to our current state.

Copy Considerations:

• Instead of stating what conformance levels are not good for, explain what would be a better measure.
• Ensure the language used does not imply that conformance levels are irrelevant.

My Updated Copy:

The severity and prioritization of accessibility issues should not be determined solely by WCAG conformance levels, but rather by the criticality of the content and how real users perceive and operate the given site.

Note: This is still open for revision, and I welcome others to contribute their versions.

[kiara-stewart]

RE: Separating priority and severity

Aren't content, perceivability, and operability crucial for accurately assessing both severity and priority? If not, it might be more effective to identify a common denominator rather than prescribing individual assessment methods for practitioners. (E.g. assign severity by taking these steps, but determine priority by following these steps)

Or, we can just address the one most likely to be misunderstood - severity (as stated in the original ask).

[Helixopp]

To be clear, since I filed the issue:

The ask is to publish an errata, or disclaimer, that conformance levels are not indicative of accessibility issue severity or priority.

Reason: the reason is because conformance levels were not assigned based on any specific parameters or formula. Instead they were just decided by consensus. They were never assigned as a matter of severity or priority.

The point of this issue isn’t to come up with a definition for conformance levels or an explanation of how they should be treated. It’s too late for that

I believe the Working Group has already agreed to the errata/disclaimer resolution. We are just formalizing the wording.

It would be inappropriate to list specific business names. Many participants in the Working Group already agree that this misconception has been observed throughout their careers. Sent from my iPhone

[Note: @mbgower removed the thread of the prior comments, which was appended to this comment, because it was triggered as an email response.]

[kiara-stewart]

Thank you for clarifying your request @Helixopp.

I understand and respect that associating a business name with an issue like this could be harmful, even if the issue is widespread.

I hope you can also respect that addressing problems is easier when we can discuss specific examples.

Since you're familiar with the businesses and initiated the original issue, could you share how you might write the disclaimer?

Side note: Whenever possible, I would appreciate it if we used specific numbers instead of quantifiers like "many," "few," or "some" to clarify our points. While this may not always be feasible, it really helps eliminate vagueness and ensure accuracy/accountability.

[Helixopp]

Please review the thread. Examples have been provided. Suggested wording has been provided. I have responded that I would like to horizontally review the suggested wording with the Accessible Platform Architectures Working Group before we commit to the text. It will take a few weeks before we can calendar this discussion on APA's agenda.

It doesn't matter how many businesses apply conformance levels this way. It matters that the text of WCAG makes this possible. The W3C doesn't hold anyone accountable for how they use our publications.

[kiara-stewart]

Suggested wording has been provided.

I know because I have authored several of the iterations myself.

I have responded that I would like to horizontally review the suggested wording with the Accessible Platform Architectures Working Group before we commit to the text. It will take a few weeks before we can calendar this discussion on APA's agenda.

I specifically requested your version of the disclaimer copy because I genuinely valued your unique context and perspective @Helixopp .

However, based on your response, I'm assuming that any comments made after your request to send this copy to the APA team have not been and will not be considered.

If my assumption is incorrect, could you clarify which version of the text we are planning to present to them?

It doesn't matter how many businesses apply conformance levels this way. It matters that the text of WCAG makes this possible. The W3C doesn't hold anyone accountable for how they use our publications.

I'm not entirely sure what you meant in this part of your message. To clarify my earlier "side note," I was requesting specific figures for use in our working group discussions.

[mbgower]

@Helixopp

I believe the Working Group has already agreed to the errata/disclaimer resolution. We are just formalizing the wording.

Do you mean that you believe there is general consensus in the comments of this issue? I think it would be more accurate to say that several suggestions have been offered by different people as ways to try to address various comments you have made. But I haven't actually seen suggested wording from you, which could really help bring this issue to some kind of completion.

Please note that the WCAG 2 Task Force has a process of vetting PRs and responses, before sending those on to the Working Group. The Working Group has not addressed this in the past to best of my knowledge. If it has, and you mean there is already a resolution to address this, please point to minutes of that resolution.

It would be inappropriate to list specific business names. Many participants in the Working Group already agree that this misconception has been observed throughout their careers.

One can provide real world examples of situations without 'naming names.' I'd welcome those. That said, if someone has done this in a public VPAT or ACR, then I think it is appropriate to cite those.

I haven't seen anyone in this thread agree that a misconception about levels by a page owner has led to significant problems with severity and priority ranking, as you describe. That's why I keep asking for details and examples. Repeatedly responding that it happens does not actually make it any more true.

[electronicwoft]

My point is clearly defining what conformance levels mean, not how people use them to prioritize issues. The fact that people use conformance levels to prioritize issues reinforces the importance of clearly articulating uniform parameters for each conformance level and consistently applying them

priority or the order in which defect A is resolved relative to defect B is meaningless without another metric to hash it with something suchb as frequency or the number of people affected - it is no less or more subjective than the assignment of severity.

Similarly, the notion that developers are somehow swayed by higher or lower measurements of severity are without evidence.

In the vast majority of situations, severity is defined for functional/systems testing, and priority is determined by product owners in a defect triage meeting or sprint ritual.

The proposed wording is an addendum rather than an erratum as it seeks to clarify rather than correct, but either way I don't believe it will prevent people from drawing comparisons.

A preferred approach might be to include an addendum that:

1. explains why traditional severity and/or priority definitions are completely unsuitable for accessibility defects, and;
2. encourages the use of levels of conformance as the basis for a meaningful resolution order that ensures desired conformance objectives can be achieved

FWIW, a normative update to WCAG 2.3 tabling the parameters used and describing how consensus was reached for each success criteria would likely assist practitioners explain the W3's thinking.

IMHO, it's the pushback from various stakeholders that is driving the perceived need for decisions about definitions to justify the relative importance of a given defect.

It is not too late.

It is very easy, for example, for a manaager to say that there is no such thing as a S1 or P1 accessibility defect because of the disconnect between functional/systems testing definitions of severity and priority and levels of conformance.

By providing an explanation in black and white, at least practitioners have something to point to when they're being railroaded by indifferent management.

From where I sit, the early draft of the WCAG 3.0 conformance model is only going to make this mismatch between specification and the expectations of so-called business stakeholders worse - if the relatively straightforward WCAG 2.x conformance model is being translated into what is more familiar, then this all-new conformance model witll fail utterly.

[electronicwoft]

FWIW, I agree with @Helixopp that there should at least be a very broad definition for what constitutes a Level A, AA, AAA SC - along similar lines as the broad WCAG 1.0 priority definitions, as currently that's not properly spelled out.

an explanation is always better than no explanation ... there's never going to be numerically verifiable threshholds or objective measurements of user impact largely because of the nature of the subject matter, but it wouldn't be too hard to slap together a table using factors like populations of affected users in terms of broadest, broader, and broad, for example, or is assistive technology required to work around a failure of this success criterion: yes, sometimes; never etc. Even an overlay of Schedule B in EN501.349 would provide practitioners with something in black and white to back-in a priority decision about a given failure. As it stands, the Easter-island statue silence in WCAG 2.x serves no-one.

[GreggVan]

It is very straight forward

Definition of a Level A (or AA or AAA ) provision

• a provision that was placed in Level A by the working group

If you want to know how they decide which level

• the decide the level by consensus.

If you want to know what criteria they used

• the nature of consensus is that everyone does not need to agree on their rationale - they just have to agree on the outcome. There are many lists (see the Understanding WCAG 2.0 for one) of SOME of the factors that were considered by the members of the working group. But there is no formula for determining it. The working group for WCAG 2.0 tried very hard to capture exactly what the rationales were and we found that there were no hard and fast rules - and the different people had different rationales. So don't look for a formula. Every time we came up with one -- we found it did not fit the results -- and the group did not want to change the their judgement on where something would go to fit any formula.

So don't look for a formula or critical factor.

Oh I can say that there was never a weighing of importance of one disability over another (except in effort -- we spent much more effort working on provisions for groups like cognitive, language, and learning disabilities since they were the hardest to find solutions for - since mostly we solved accessibility using AT (there are very few provisions for people who are blind -- but they had AT (Screen readers) so they could make materials accessible themselves as long as the information was compatible with AT. But for cognitive, language, and learning disabilities there was essentially no AT so that group could not make thing accessible for themselves like blind people could -- but just using the AT compatibility provisions. There are therefore many more provisions for direct access (no AT) for people with cognitive, language, and learning disabilities than for any other disability including blindness. In fact - we tried to get everything we could think of that would qualify (be testable) into WCAG 2.0. There never was a time when we excluded or downleveled a provision because of disability.

Best g

[electronicwoft]

There is no question as to the extraordinary efforts of those involved in formulating what has pioneered an entire discipline - this is not about saying the original working group did something fundamentally wrong. Nor is it to say that WCAG 2.0 is no longer entirely fit for purpose.

But, in my experience, just restating that this is the consensus a group of people arrived at for a given success criterion two decades earlier when being questioned by an indifferent and likely ignorant stakeholder is just not very persuasive when prosecuting a case for why this or that carries greater or lesser weight than something else.

Noone could have predicted the way that WCAG 2.0 has been desseminated, applied, paraphrased, interpreted, or challenged no more than the scribblers of various holy scriptures could be thousands of years ago.

I am sure you can see the clumsy parallel ...

So, while there may not be a 'formula or a critical factor', a working group note fleshing out the explanation in the latter part of your comment with further explanation of what levels of conformance are not equivalent to or should not be correlated with, or addressing the 'churches' which have evolved with the specification would be - in my view, at least - more helpful than what is published now about how success criteria are assigned to levels of conformance. > It is very straight forward

[GreggVan]

The explanations are already documented. We took great care to do that and mention all the major factors that were involved. There is not any more to say. There were a myriad of factors. But the strongest ones are already listed.

when people ask for more -- they ask for determinants -- which were the "criteria" for being A or AA -- as if there was a formula or weighted list. Or that everyone used the same criteria for their decision as to which level. They didn't. We had advocates for every disability and sub-disability, from industry (and they had different views and priorities and thoughts ) and academics and developers and government and evaluators and just volunteers. Together they took each provision -- discussed at great length - and decided which level it would go in. Then they debated back and forth until they reached consensus on a level for each one.

So that is why I said that the only definition of exactly why each one ended at a particular level is -- that they were put at the level that the working group reached consensus that it should be put at. Lots of people with lots of views on lots of factors -- and probably few if any people that put the same weight on the same factors in making their decision.

So there were no rules or defining factors or formulas for deciding. So the best that can be done is to describe the key factors that people said they used (which is done in WCAG/Understanding WCAG). And to say that the final decision was the decision made by the group.

Hopefully that makes it clearer.
The reply is not meant to be dismissive of the desire to know more. But that the question has been answered as best as it can be and still be accurate.

best

[electronicwoft]

thank you for taking the time elaborating how it went down back then, but, for me and presumably others, what is in the understanding documents just doesn't cut the mustard anymore for practitioners at the coalface so I guess I'll leave it there ...

[patrickhlauke]

"why is this SC in level FooBar?" "because the group decided that it should go in FooBar" "so what is the meaning of level FooBar?" "it's the level that the group decided SCs like this one should be in"

it keep reminding me of: “When I use a word,’ Humpty Dumpty said in rather a scornful tone, ‘it means just what I choose it to mean — neither more nor less.’"

and to be clear, we're not asking for a definitive formula here, but clearly, the parameters/axes that led the group to a particular decision have been laid out. things like (but not limited to) "how severe the problem is for certain user groups", "how many user groups are affected", "how easy/hard it is to achieve a better outcome/implement a solution", "are there workarounds that make the problem circumventable".

so clearly the axes have been defined, just that nowhere it has stated, in a simple way, how those axes influenced a decision: when an SC tended to affect more users, was easier to implement, and had no workarounds, it generally ended up in Level A; on the other end of the spectrum, if an SC tended to not affect as many users, or was harder to implement, or there were workaround (albeit laborious/convoluted) that made them less of a showstopper, then an SC tended to be lumped more towards Level AA or AAA. this seems fairly non-controversial statement (though of course not elegant, but far from a "definitive formula" which nobody has actually been asking for), but somehow there's a strong reluctance to acknowledge this for some reason...

[JAWS-test]

I've read all the comments, but there are so many now that I'm not sure if I've missed something:

• there is a definition for A, AA, AAA
• the definition is not satisfying for some because it is not precise enough.

However, a better definition is not possible because the SC levels are already assigned and even if we were to come up with a new definition, the existing levels would no longer fit. But they can't be changed (so easily). Therefore: Why not live with the fact that there is a definition that may not be perfect, but is still comprehensible. @patrickhlauke: The 3 levels are not just Humpty Dumpty ...