Closed DavidMacDonald closed 3 years ago
@DavidMacDonald: I appreciate your concerns and I think we have some confusion of language. "Testability" and "Measureability" don't have an intrinsic difference in meaning. We are using that language to draw a distinction between the WCAG 2.x reliance on statements that can be evaluated with a pass/fail test, and the Silver direction to allow a more broad way of testing, that can still be used in a regulatory environment. Some of the success criteria from COGA and Low Vision task forces were deferred to Silver because they didn't fit the WCAG 2.x structure of testable statements. Therefore, Silver has to have a new structure that will include the needs of people with disabilities that do not fit into WCAG 2.x.
The Silver Design Sprint involved lawyers, policy makers and regulators. The idea for the Conformance model that we are currently developing came from an accessibility-specialist lawyer in the US Office of Civil Rights. As we continue to develop the Silver prototypes, we will be user testing them with a number of people, including lawyers.
We need to do this to serve the needs of people with disabilities who were not served by WCAG 2.1. That's why it's a Requirement. We have a pretty good idea of how to do it, we already have experts (including lawyers) who have taken a critical look at the basic ideas, and we feel comfortable that we can make it a requirement.
@jspellman
We are using that language to draw a distinction between the WCAG 2.x reliance on statements that can be evaluated with a pass/fail test, and the Silver direction to allow a more broad way of testing, that can still be used in a regulatory environment.
I agree that this is the intention of the Silver TF, and it is grounded in reputable recommendations. However, we don't have consensus at this point and have not ever discussed formally on the WG this foundational change in the evaluation of accessibility, and yet this requirements document is presenting it like it has been decided by the WG.
The Silver Design Sprint involved lawyers, policy makers and regulators.
I think that is amazing, but it still has never been discussed by the working group. A new conformance model has to be discussed, not just implemented in the Requirements doc as a fiat. I think the requirements document should say something like:
"a thorough investigation will be conducted into widening the evaluation of the guidelines to include various models that would bring to play other considerations beyond pass/fail testing under a new term, measureability"
I agree that this is the intention of the Silver TF, and it is grounded in reputable recommendations. However, we don't have consensus at this point and have not ever discussed formally on the WG this foundational change in the evaluation of accessibility, and yet this requirements document is presenting it like it has been decided by the WG.
To that point, we specifically brought the Requirements draft to the Working Group with the expressed purpose of working to consensus on the points within it. Not just on the requirement around the conformance model, but all of the requirements included in the draft (and to identify requirements we might've left out!).
A new conformance model has to be discussed, not just implemented in the Requirements doc as a fiat.
Very much agreed! We haven't proposed a new conformance model, but we do know through the research that we've done that we need something different than the current conformance model's strict pass/fail requirement in order to (among other things) include guidelines around some needs of people with disabilities that can't go into WCAG because of it.
Through the research and prototyping that we've done so far, we've found that measurability addresses several things in one shift, but we haven't yet done the amount of prototyping and testing to know exactly what that would look like in terms of a conformance model. Before we do that work, though, we want to get consensus that we need to do that work, if that makes sense.
To that point, we specifically brought the Requirements draft to the Working Group with the expressed purpose of working to consensus on the points within it.
I don't think the requirements doc is the proper context for this huge shift. I think that research and consensus building is a result of our work, not the beginning premise for it. If the TF has started that process on our behalf, that is amazing. We all want to find a viable way forward that builds on the success of WCAG.
The AGWG should look carefully at the research and the statements of the lawyers, and contact them for further clarification and articulation so that we thoroughly understand the issues and the proposals. This kind of a fundamental change is not something that should be decided by the AGWG over a 2 week CFC which marks the *beginning of our work" on the next major version.
My understanding of the job of a TF is to aid the WG in its mandate. This paragraph seems to turn it the other way around. My proposed change is something like:
"a thorough investigation will be conducted into widening the evaluation of the guidelines to include various models that would bring to play other considerations beyond pass/fail testing under a new term, measureability"
Hi @DavidMacDonald,
Sorry for the duplication, I'd like to reply publically as well...
We (the AGWG/W3C) setup Silver to: “perform preliminary development of a new version of Accessibility Guidelines following a research-focused, user-centered design methodology to produce the most effective and flexible outcome.”
The requirements document isn't proposing a definate shift, it is reflecting the requirements that have come from the research and outlining potential approaches.
The "Measurable Guidance" bit is under the heading "Opportunities", it is a possible avenue for the guidelines which needs to be explored.
True/false is one measurement (and Silver will likely include the majority of WCAG 2.x as a starting point), this is opening it up for other methods of testing. I don't see it as removing 'testable', but allowing for more methods of testing.
The prototypes will attempt to meet the requirements, i.e. try to build in an SC which uses measurability. It doesn’t say that none of the SCs will be testable, it is allowing for (some) SCs to be measurable. (And I’d like to allow for some that can be met by saying you’ve done a particular process, see pull request #24).
Multiple models / prototypes will be created tested, discussed. If you would like to create a prototype that tries to meet the requirements (e.g. covering COGA SCs) that only uses ‘testable’, please do join the Silver TF. It is both a TF and a community group for maxumum participation.
However I would note that in general, a consensus-process is not useful when you are exploring various ideas in a divergant fashion. That will be useful when it comes to narrowing down the ideas to what will work best, but we are not at that stage yet.
Cheers,
-Alastair
What happens if after investigation that for some reason we can't make the wider conformance models work? Are we meeting our requirements doc if we have a conformance section that says "testable" pass/fail statements without wider means of measuring conformance. Are we now committed to going with the term "measurable" in the final, or in the first iteration of Silver, and are prevented from staying with "testable"?
Hi @ alastc and all,
We (the AGWG/W3C) setup Silver to: “perform preliminary development of a new version of Accessibility Guidelines following a research-focused, user-centered design methodology to produce the most effective and flexible outcome.”
The requirements document isn't proposing a definate shift, it is reflecting the requirements that have come from the research and outlining potential approaches.
The charter says AG will:
…Incubate requirements for a major update to WCAG to address usability and conformance challenges…
I think that the sticking point may be the words "preliminary development" and "Incubate". We may not have a shared understanding of what those terms entail.
If it is not a "definite shift", I think something like @DavidMacDonald proposed paragraph clarifies it. David suggested:
"a thorough investigation will be conducted into widening the evaluation of the guidelines to include various models that would bring to play other considerations beyond pass/fail testing under a new term, measureability"
Then perhaps add that "Preliminary work in this regard has been initiated."
What happens if after investigation that for some reason we can't make the wider conformance models work?
Then we don't go with that. These requirements are for showing the scope and aim of the process, they do not decide the outcome.
I disagree with adding:
"a thorough investigation will be conducted into widening the evaluation of the guidelines...
Because that is what the whole process is! Not just on the point of conformance, but all the other 'opportunities' as well.
Hi Alastair,
I agree that David's paragraph should not go into the "Opportunities" section.
It is gemane to the actual "Requirements" section of the Requirements document. I assumed that would to be the most logical place for it.
David, is that the location that you had in mind?
Thanks.
On Mon, Jul 16, 2018, 4:57 PM Alastair Campbell notifications@github.com wrote:
What happens if after investigation that for some reason we can't make the wider conformance models work?
Then we don't go with that. These requirements are for showing the scope and aim of the process, they do not decide the outcome.
I disagree with adding:
"a thorough investigation will be conducted into widening the evaluation of the guidelines...
Because that is what the whole process is! Not just on the point of conformance, but all the other 'opportunities' as well.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/w3c/silver/issues/22#issuecomment-405394455, or mute the thread https://github.com/notifications/unsubscribe-auth/AF7TkONChKJg4imPyT0_bBjZsmjlHbbLks5uHQxXgaJpZM4VJrOL .
I believe @jspellman has in mind to do an overview of the process, to answer this and other questions along those lines. That would be a more suitable place, and this doc can be the requirements / scope. We can pick it up on the call tomorrow.
David, is that the location that you had in mind?
I would integrate it into the bullet point on Measurability. It would like something like this:
1.2.2 Conformance Model Investigation into Measurable Guidance: Certain accessibility guidance is quite clear and testable. Others, far less so. There are needs of people with disabilities, especially cognitive and low vision disabilities that are not well served by guidance that can only be tested by a pass/fail statement. A thorough investigation will be conducted into widening the evaluation of the guidelines to include various models that would bring to play other considerations beyond pass/fail testing under a new term, measureability. Multiple means of measurement may allow inclusion of more accessibility guidance.
Hi @DavidMacDonald and all,
I think your latest proposed verbiage is a good compromise and could fit into the "Opportunities" section.
Given that, perhaps we should consider placing a note in the "3.1 Multiple ways to measure" section. That section lists the actual Requirements. Maybe something such as the following could work:
Editor's note:
Preliminary research indicates that different guidance has potential for different measurement. Silver is pursuing other solutions at this time. For example, guidance could possibly describe "minimal" and "better" methods of conforming to a requirement or allow user testing and usability testing as a method of conformance. We will relook at this requirement in a future version of this document.
From the call today we were told:
I think this issue should stay open until the WG formally makes a decision that this is the way the Guidelines will proceed.
PS I think there are probably 2 changes to the paragraph that better reflect the intention of the Silver TF.
Measurable Guidance: Certain accessibility guidance is quite clear and
<remove>measurable</remove><add>testable</add>
. Others, far less so. There are needs of people with disabilities, especially cognitive and low vision disabilities that are not well served by guidance that can only be<remove>measured</remove><add>tested</add>
by pass/fail statement. Multiple means of measurement allow inclusion of more accessibility guidance. https://w3c.github.io/silver/requirements/index.html
Hi David,
I think there is a confusion between 'testable' and the type of testable we are used to in WCAG 2.x
Something which is measurable is by definition testable. If you can measure it, that is a form of test.
What we're getting at is that only using binary answers from testing (true/false) is not sufficient across the board.
I agree that the paragraph doesn't quite make that clear though, how about:
Measurable Guidance: Certain accessibility guidance can be tested in a binary fashion, where a page either meets the requirement or does not meet it. However, there are some needs of people with disabilities, especially cognitive and low vision disabilities that are not well served by guidance that can only be measured by pass/fail statement. Multiple means of measurement allow inclusion of more accessibility guidance.
Something which is measurable is by definition testable. If you can measure it, that is a form of test.
Does not true/false not correspond to pass/fail? To capture what Detlev said yesterday in the meeting.
We'll probably all agree on sites that are really bad at one end and really good at the other but there is a wide plateau in the middle which will be very hard to determine whether they pass or not.
This will be a challenge for any model that doesn't have a true/false set of tests, and I'll be interested in proposals that are advanced to overcome this.
An Example of this is:
Does not true/false not correspond to pass/fail?
It does now, but when you expand that to include measurable, a pass could also be:
Those are just ideas, the point is that true/false statements are only one possible method of passing, that’s kinda the point here.
The requirement for measurability in this doc means it is in scope for exploration and testing, not that there is a foregone conclusion.
90 or more out of 100 terms used are plain language.
would not the true statement be over 90 and false be under 90? (assuming the group could agree on what "plain" means)
8 / 10 score for readability.
the true statement would be 8 or more, false would be 7 or less. (assuming there is a score our there that works across languages???)
For alt text, that all instances on a ‘critical path’ pass, but not necessarily all on the page.
I hope we don't relax our alt text requirements. There has been reasonably unity about alt over the years and not much push back from industry.
That you have done an applicability assessment, and decided and documented that usability testing needs to be done.
"True" would be that it is documented, "false" is that it hasn't been documented.
One strategy to address issues that are difficult to test, could be that for SCs (or whatever they are called) that can't meet testability conformance are documented and published publicly linked from the home home page.
So the accessibility link at the bottom of the home page would go to a page that has statements something like this.
Here are the steps we took to simplify the language on this site
The pass/fail condition is based on whether the statement is posted .... from there we can let public scrutiny be the determining factor.
Hi @alastc and all,
I believe @jspellman has in mind to do an overview of the process, to answer this and other questions along those lines.
That would be helpful. I ran across the document, "Draft of Silver relationship to W3C", in the Silver Google Drive.
But further clarification of process and relationships would be appreciated.
I think we're just getting down to a language issue now, the examples above were just spit-balling, I wasn't saying we should change alt text.
The current SC are statements about the content that you answer as pass or fail. If we use a score (numbers, percentage etc.) you could build that into the statements and force it into a true/false approach, but you'd be missing an opportunity.
There is room for exploration of other methods. For example, if there were a bunch of SCs that are scored out of 10, you could then take an average and pass/fail on that basis.
The point is that we are not saying criteria are not testable, but we are exploring other options in how that reaches a pass/fail.
Secondarily on the process-approach point, security is a good parallel in terms of approach, and the ISO 27001 could provide some clues here. (I'm not saying we re-create that, it is large and difficult to acheive, but it might provide some methods we can try.) For example in ISO 27001 the bits that caught my attention were that you:
Anyone familiar with ISO 27001 will be able to poke holes or say that isn't quite right, I'm still investigating it with a new collegue who's been through it a couple of times.
However, that kind of framework would provide a way to include certain user-centred-design approaches such as usability testing & IA methods which would be very helpful for including COGA requirments and testability.
Anyway, the requirment is to try including SCs that are measurable, is there an objection to that now?
I think we need language built in to allow us to back track to testability if necessary, rather than saying
"we have to make non testable models work, its in our requirements doc."
Particularly during transition releases of silver (after 2.x), of which there may be several with this 18 month dev cycle. If not, then I fall back to the assurances given during our meeting.
I think this issue should stay open until the WG formally makes a decision that this is the way the Guidelines will proceed.
Agreed.
Regarding process, it would seem that until the AG Working Group makes a decision to accept the document as the AG Requirements for Silver, the doc remains a "Draft Community Group Report" as the subtitle of the current doc states. Is that correct?
Thanks, Laura
On Sun, Jul 22, 2018, 6:00 AM David MacDonald notifications@github.com wrote:
I think we need language built in to allow us to back track to testability if necessary, rather than saying
"we have to make non testable models work, its in our requirements doc."
Particularly during transition releases of silver (after 2.x), of which there may be several with this 18 month dev cycle. If not, then I fall back to the assurances given during our meeting.
- This is a first draft of the requirements document
- It will not be finalized for about a year
- There will be at least 2 more drafts
- If measurability doesn't work out due to industry pushback, implementation problems, legal complications, or evaluation problems, etc., we are not committing to measurability and we can return to testability without violating any previous decision, because a decision has not been made.
I think this issue should stay open until the WG formally makes a decision that this is the way the Guidelines will proceed.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/w3c/silver/issues/22#issuecomment-406857556, or mute the thread https://github.com/notifications/unsubscribe-auth/AF7TkIJq0F0nLlakaIZNVQPuoZqP9kSWks5uJFtLgaJpZM4VJrOL .
Hi @jspellman ,
- Silver is not…
- Silver wants…
- Silver is not…
- Silver has…
When you speak of "Silver" do you mean the Silver Community Group Participants and Silver Task Force Participants?
Thanks, Laura
Hi @lauracarlson , we are running them as one group. We share the email list, the wiki, the action items, and the conference calls. The items I listed are a consensus of the active members of the joint group.
Thank you, Jeanne.
As discussed in the Silver meeting of 4 December 2020, this thread will be closed as being overly broad and then overtaken by events. If there are specifics that still need to be addressed, please start a new specific thread.
This is a fundamental shift in the basis of Guidelines. There is no mention of testability in the Silver Requirements document and I understand that the direction of Silver is looking to shift from testable to measurable. I don't think we are ready to make that kind of assurance in this requirements document.
There is no detailed discussion of this fundamental shift in the document. This may or may not be a good thing. It could work out very well or it could make the guidelines unusable in a legal framework.
We would need to get input from lawyers who have been involved in multiple settlements and lawsuits involving WCAG. In my work with large companies I've had exposure to some of these lawyers, and could pull them into this conversation. I expect others in the group have access to lawyers who have litigated around WCAG.
My point is that it is premature to declare this shift for Silver in the requirements document, but rather it should say that its a requirement to investigate testability vs measurability as a basis for the success criteria (or whatever they are called).