Evaluation teams is not realistic

[DavidMacDonald]

I'd like to suggest that the following statement is problematic, and that it be readjusted.

"Use combined expertise: Evaluating web accessibility requires diverse kinds of skills and expertise. For example, some requirements relate to the design, writing, and development aspects of a website, while others relate to assistive technologies and their use by people with disabilities. Sharing evaluation tasks in a team of reviewers can help make evaluation more effective and efficient. Learn more" http://www.w3.org/WAI/eval/reviewteams

We had long discussions about this on the Evaluation Methodology task force. Here are the issues:

• The statement puts unfair emphasis on multi member teams and large consultancy houses, and disadvantages accessibility consultants. It give the perception that a large accessibility house provide better coverage than a knowledgeable consultant.
• Even the large accessibility houses assign one assessor to a project. Even if they have 25-50 accessibility testers, they don't assign multiple assessors to a project. It's not practical and doesn't usually provide better results.
• The link provided goes to an article that is over 15 years old. It is dated and is not how the industry has developed.
• Note: Of course usability testing with people with disabilities with AT is important, and should be conducted but I don't think we should conflate that with a conformance audit, although it is often incorporated into an audit. I think that could be included as a separate bullet point.

Here is the language we used in the Evaluation Task force:

Combined Expertise (Optional) This methodology can be carried out by an individual evaluator with the skills described in the previous section (Required Expertise), or a team of evaluators with collective expertise. Using the combined expertise of different evaluators may sometimes be necessary or beneficial when one evaluator alone does not possess all of the required expertise. ...

[James-Green]

+1 to having this discussion. I would add that since we are making efforts to not scare people new to accessibility, telling them that "Evaluating web accessibility requires diverse kinds of skills and expertise" and saying we suggest they involve a bunch of people may hurt our cause.