w3c / silver

Accessibility Guidelines "Silver"
https://w3c.github.io/silver/
Other
198 stars 44 forks source link

Peer review of color contrast research #272

Open mfairchild365 opened 3 years ago

mfairchild365 commented 3 years ago

The work, research, and changes to the color contrast formula is an amazing step in the right direction from what I can tell.

However, I'm not a color scientist and I honestly don't understand the science behind it. I can't find anything regarding whether or not the research has been peer reviewed by other color scientists. This raises further questions and concerns for me because this sort of change could have vast ramifications into the future.

Questions that I'd like answered:

Again, I'm pleased to see progress here, but I'd feel much more comfortable if I could answer those questions.

Myndex commented 3 years ago

Hi Michael @mfairchild365, thank you for the comment. You know there is a famous color scientist named Mark Fairchild.

The work, research, and changes to the color contrast formula is an amazing step in the right direction from what I can tell.

Thank you

Thank you, this is a multi-year project, and we are not even at the mid-point.

However, I'm not a color scientist and I honestly don't understand the science behind it. I can't find anything regarding whether or not the research has been peer reviewed by other color scientists. This raises further questions and concerns for me because this sort of change could have vast ramifications into the future.

All important considerations, and yes I know color science is a VERY difficult subject, abstract, and not well understood by most — the fact we are taught incorrect things (like primary colors) early in life does not help!!

Solid Science and Links

Everything I am doing is based on a solid foundation of science, including modern appearance models such as the Hunt model, R-LAB, and others, and much more.

Here are the basic links, several including extensive bibliographies, and among these links are dozens of further links covering the information you are asking for.

W3 Wiki (essential whitepaper materials) This includes an extensive bibliography.

Discussion of Experiment CE_17 While this was over a year ago, it is a good summation of the general principals involved. Also at this same site are other experimental results that you can review for yourself.

Fonts for Accessibility A review of several dozen fonts discussing the accessibility implications.

Gist on the "Orange Conundrum" A demonstration of why the old math has a high error rate.

Issue #695 on the WCAG github, that started this journey. I am on record regarding the underlying issues of the current 1.4.3.

CVD Simulator This is an app I created that simulates Color Vision Deficiency (often incorrectly called "color blind").

SAPC Development and Research Site This is the public facing beta of SAPC and the APCA algorithm. Here, there is a "research mode" with interactive experiments you can examine for yourself.

Evaluation and Evidence

_Questions that I'd like answered:

  • Has the research and outcomes been peer reviewed? If yes, by who and what are their credentials? If no, when will it be peer reviewed?_

All my work is grounded in a solid foundation of modern vision science. See the bibliographies. In addition there has been frequent informal peer review and discussion. If you are asking about peer-reviewed publications, you may not be aware, but the author does not get paid — they have to pay, and it can be substantial. W3 is a non-profit, and there is no such budget allowed.

There are patents applied for or in progress thereof. I do have multiple papers I am working on for peer reviewed publication, at my own expense. This is a process that can take years (I've heard of researchers waiting 3 or more years for publication). As such it is impossible to quote you any date. And peer review boards are typically anonymous.

Regardless, On several of the associated links above I provide prima facie evidence, and you don't need to be a color scientist to grasp them — it is plain as day to see, in particular the side by side comparisons.

MOREOVER: peer review does not mean that the efficacy of something is valid. It is trivial to point to peer reviewed published materials that were found to be utterly bogus. The value, while of some importance, should not be over-emphasized.

  • Was the research tested with actual humans? If so, what was the makeup of the sample, sample size, and research methodology? If no, will this happen?

Yes of course, human test subjects have been involved from the start. COVID put a damper on the larger planned mass-public study, now set for next summer. Many experimental results and discussions are on the above links. Explore. I've been fairly open about the process.

* Is the research reproducible?

I have provided links that include interactive experiments that you can evaluate in the privacy and comfort of your own home. The prima facie evidence for you to evaluate is right there in the viewable, easy to use, and reproducible results.

Evidence in Solution

(No stirring needed)

Again, I'm pleased to see progress here, but I'd feel much more comfortable if I could answer those questions.

If it helps you to grasp this, the current WCAG 2 contrast spec was not peer reviewed research, and otherwise was based in-priori, in some places citing incorrect references, including some incorrect math, and there have been papers by others written on this problem. As I noted above, I started GitHub issue #695 nearly two years ago pointing out the problems and subsequently set forth to find a real, correct solution.

A solution many have been waiting for. But the subject matter is dense, abstract, and complex — and in some areas of the art there is a lack of consensus, and active research is still undergoing and we are still learning new aspects of visual perception and readability every year.

ME: I am a Hollywood film & television industry professional. I could say that I have half a century of experience in imaging, visual perception, light, optics, etc. (that's If I count I started in photography at age 8, made my first short film at age 10, LOL) Seriously though, this field is my life long passion, with nearly four decades of professional experience in broadcast engineering, film and television production, human perception & related technologies that extend to typography, print, and design. I am a three time Emmy® award winner and my on-screen credits are a matter of public record on IMDB. I was here in Hollywood during the transition from chemical imaging (film) to digital, and the growing pains and the expansion of knowledge that ensued in this area, published articles on work flow, best practices, and so forth.

I lost my eyesight due to early onset cataracts, which has only been partially restored through six surgeries. A bit over two years ago while recovering, I decided to write a book on modern color theory which led to the discovery of the inherent problems with the current 1.4.3 standard. That led to issue #695, and the present research project.

I hope this answers your questions.

Thank you,

Andy

Andrew Somers W3 AGWG Invited Expert Myndex Color Science Researcher Inventor SAPC and the APCA

mfairchild365 commented 3 years ago

Thank you Andy for your reply, hard work, and contributions.

I’d still like to see some 3rd party validation of the research and outcomes. I’d also like to hear other folks perspectives on my original questions.

patrickhlauke commented 3 years ago

maybe his "faux" concern, as you put it, is merely a desire not to enshrine something in a spec that, despite all the care and goodwill put into it, may have a flaw? instead of immediately presuming bad faith and shadowy conspiracy motivations...

mfairchild365 commented 3 years ago

maybe his "faux" concern, as you put it, is merely a desire not to enshrine something in a spec that, despite all the care and goodwill put into it, may have a flaw? instead of immediately presuming bad faith and shadowy conspiracy motivations...

This sums up my concern pretty well. I recognize the flaws of WCAG 2.x and the need for change, especially in this area. In fact, I'm very optimistic about the changes in this area. I also know that even the most careful and well intentioned work can still have flaws. Given the potential for international application, legal adoption, and most importantly the impact on the very diverse set of people that is humanity, I firmly believe that procedures should be in place to vet research and outcomes in a way that is not biased and avoids potential conflicts of interest.

I also disagree with the notion that this is now my personal problem to fix. I, personally, do not carry the burden of proof for the research and outcomes used in this draft spec, even if I truly hope that they are accurate and will result in the change that I want to see.

alastc commented 3 years ago

@mfairchild365 - there are links above that are worth reviewing with regard to what the new approach is based on. There will also be a fairly long process of reviews for all the new guidelines over the next few years.

This is not research the W3C itself would undertake because it is a membership organisation. But (given the need) it may be something that some of the members would undertake in due course.

@Myndex - people are going to ask all sorts of questions with various assumptions, please do not assume it is all on you to answer these questions. The original response is useful, and I suspect it would make a good basis for an FAQ section somewhere that we can point other people to who have the same question.

We often get asked for evidence & research across many of the guidelines. It is not unusual and should not prompt suspicion.

All that needs to happen is that we take this issue as a request for 3rd party review. If necessary, the TF facilitators / chairs can discuss and reach out to people in a position to help.

jspellman commented 3 years ago

Thank you for your comment. Project members are working on your comment. You may see discussion in the comment thread and we may ask for additional information as we work on it. We will mark the official response when we are finished and close the issue.

ChrisLoiselle commented 3 years ago

@jspellman This is ready for survey. A call to third party reviewers would benefit , however would not prohibit current use of tool within WCAG 3.0 FPWD. As stated in #370 , additional research studies will be completed by Andy after COVID has passed. If a third party wanted to volunteer to review, they are welcome to do so.

ChrisLoiselle commented 3 years ago

DRAFT RESPONSE: A call to third party reviewers would benefit , however would not prohibit current use of tool within WCAG 3.0 FPWD. As stated in #370 , additional research studies will be completed by Andy after COVID has passed. If a third party wanted to volunteer to review, they are welcome to do so.

I hope this answers your question, if it does not, please feel free to follow-up.

Thank you, Chris

joelanman commented 2 years ago

Was the research tested with actual humans? If so, what was the makeup of the sample, sample size, and research methodology? If no, will this happen?

I think this is extremely important, and that the research includes disabled people with a broad range of sight issues. Apparently this is underway but has not yet been published. I wonder if there are ways we can contribute research as a community. It should be possible to do remotely during the pandemic.

Myndex commented 2 years ago

Hi Joe @joelanman

I think this is extremely important, and that the research includes disabled people with a broad range of sight issues. Apparently this is underway but has not yet been published. I wonder if there are ways we can contribute research as a community. It should be possible to do remotely during the pandemic.

The research thus far has been mainly confined to our labs, including pilot studies to the planned large-scale remote studies. COVID among other things has had a marked impact.

There are issues with remote studies, namely the lack of laboratory controls, and related concerns of obstructive outliers. Both should be quelled to some degree with a large enough sample size.

The software for conducting this is not yet complete. The user ID and management system still needs some work, and the key experiments themselves, while they work well in a laboratory setting with a facilitator, need to be automated for a large sample remote study.

These topics are up for discussion at the main APCA/SAPC GitHub repo.

It is useful to point out that the AGWG and W3 do not perform research — research is independently performed.

The main repo for this discussion is: https://github.com/Myndex/SAPC-APCA/discussions

HELP WANTED: it has become obvious to me that conducting good research, AND supporting applications and npm releases AND authoring guidelines AND developing study software is, uh, a lot... The research is at the point now where additional help might be useful... There will be a "help wanted" thread in APCA discussions later this week.

Thank you,

Andy

Andrew Somers
W3 AGWG Invited Expert
Myndex Color Science Researcher
Inventor SAPC and the APCA

THE WORLD IS READING

Myndex commented 2 years ago

I believe there is a fairly massive misunderstanding regarding the nature of peer review, and apparently I am not capable of describing the process & status in a post in a way that is understandable. I.e., I'm not getting the facts across.

The underlying science that APCA guidelines reference is all well established, peer reviewed, scientific consensus.

In an effort to clarify, I've created a catalog of resources relating to APCA guidelines, so that everything is all in one place: https://git.myndex.com

APCA documentation and resources -  the world is reading

To be clear, this is not to imply that there won't be further review and publications, but there seems to have been a disconnect from the materials and bibliographies I've made available, and people accessing them. I don't know how else to account for the disinformation that has been propagated on the subject, as everything has been open and available since the project started over three years ago. Additional testing and studies are continuing.

Myndex commented 7 months ago

The WCAG 3 repo has an issue thread containing a list of independent third party reviews of APCA, including journal published reviews.