vincerubinetti
commented
1 week ago Okay, there's several issues here.
First, anything you pass to Playwright via page.evaluate must be serializable. That's understandable. So that's why the axe-playwright library doesn't work when passing a custom check with a custom evaluate function, because functions are not (easily/automatically) serializable. And @axe-core/playwright doesn't even provide an option to pass custom checks at all, lol.
So, I tried just basically copying the code from this library right into a page.evaluate statement. It depends on apca-w3 (which itself has dependencies) for the actual calculations, so I had to inject that library too in a vanilla-js fashion way. So I ended up with this absolute monstrosity:
code
```ts await injectAxe(page); await page.evaluate(async () => { await new Promise((resolve) => { const script = document.createElement("script"); document.body.append(script); script.type = "module"; script.src = "https://cdn.jsdelivr.net/npm/apca-w3@0.1.9/+esm"; script.addEventListener("load", resolve); }); const apcaRule: Rule = { id: `color-contrast-apca-silver`, impact: "serious", selector: "p", // matches: "color-contrast-matches", metadata: { description: `Ensures the contrast between foreground and background colors meets APCA silver level conformance minimums thresholds`, help: "Elements must meet APCA conformance minimums thresholds", helpUrl: "https://readtech.org/ARC/tests/visual-readability-contrast/?tn=criterion", }, all: [`color-contrast-apca-silver-conformance`], tags: ["apca", "wcag3", `apca-silver`], }; const apcaCheck: Check = { id: `color-contrast-apca-silver-conformance`, metadata: { impact: "serious", messages: { pass: "Element has sufficient APCA silver level lightness contrast (Lc) of ${data.apcaContrast}Lc (foreground color: ${data.fgColor}, background color: ${data.bgColor}, font size: ${data.fontSize}, font weight: ${data.fontWeight}). Expected minimum APCA contrast of ${data.apcaThreshold}}", fail: { default: "Element has insufficient APCA silver level contrast of ${data.apcaContrast}Lc (foreground color: ${data.fgColor}, background color: ${data.bgColor}, font size: ${data.fontSize}, font weight: ${data.fontWeight}). Expected minimum APCA lightness contrast of ${data.apcaThreshold}Lc", increaseFont: "Element has insufficient APCA silver level contrast of ${data.apcaContrast}Lc (foreground color: ${data.fgColor}, background color: ${data.bgColor}, font size: ${data.fontSize}, font weight: ${data.fontWeight}). Increase font size and/or font weight to meet APCA conformance minimums", }, incomplete: "Unable to determine APCA lightness contrast (Lc)", }, }, evaluate(node) { const style = window.getComputedStyle(node); const fontSize = style.getPropertyValue("font-size"); const fontWeight = style.getPropertyValue("font-weight"); const bgColor = axe.commons.color.getBackgroundColor(node); const fgColor = axe.commons.color.getForegroundColor( node, false, bgColor, ); console.log(node, bgColor, fgColor, fontSize, fontWeight); if (!bgColor || !fgColor || !fontSize || !fontWeight) return undefined; const toRGBA = (color) => `rgba(${color.red}, ${color.green}, ${color.blue}, ${color.alpha})`; const threshold = (fontSize) => { const size = parseFloat(fontSize); if (size >= 32) return 45; if (size >= 16) return 60; return 75; }; const apcaContrast = Math.abs( calcAPCA(toRGBA(fgColor), toRGBA(bgColor)), ); const apcaThreshold = threshold(fontSize); this.data({ fgColor: fgColor.toHexString(), bgColor: bgColor.toHexString(), fontSize: `${(parseFloat(fontSize) * (72 / 96)).toFixed(1)}pt (${parseFloat(fontSize)}px)`, fontWeight, apcaContrast: Math.round(apcaContrast * 100) / 100, apcaThreshold, messageKey: apcaThreshold === null ? "increaseFont" : "default", }); return apcaThreshold ? apcaContrast >= apcaThreshold : false; }, }; const config: Spec = { checks: [apcaCheck], rules: [apcaRule, { id: "color-contrast", enabled: false }], }; await window.axe.configure(config); await window.axe.run(); }); ```So, one thing that might help is if this apca-check library could export a fully bundled single js file as part of its release process, so I could just inject the whole thing into the Playwright browser like:
const apcaScriptString = readFileSync(resolve("apca-check/apca-check.min.js"));
await page.evaluate(
(scriptString) => window.eval(scriptString),
scriptString
);But, even when I did the above monstrosity, I still ran into issues. It was at least able to register the rules and checks. Running Playwright in --debug mode, and opening the browser's console after injecting the axe script and rules/checks, I could see that color-contrast-apca-silver was in axe.getRules() and axe._audit.rules and axe._audit.checks.
But I got no violations on a test where it definitely should've failed: a paragraph with white text and a light beige background. Note the console.log in my above code. It wasn't even checking the paragraph! For some reason, the matches: "color-contrast-matches" was only checking like 3 elements on the whole page. That is really concerning to me. Like, has Axe this whole time been checking only a fraction of my page elements in the standard tests?
Finally I forced it to selector: "p",, but the rule still failed to report a problem. Yes I'm using the looser bronze threshold function, but making it way more strict still did not report an error.
So....!!! I've wasted way too much time on this, so I'm just giving up. Someone else is going to have to figure this out. I'll check color contrast manually with Lighthouse whenever I have the time. Or maybe I'll try moving to the Lighthouse CLI tool.
I use https://www.npmjs.com/package/axe-playwright to run my tests, which requires you to do...
Perhaps it would be enough to export
rulesandchecksfrom this library so we could use them.Edit 1
I tried to figure out how to get the rules and checks this plugin has registered back from axe core, like
axe.getRules()oraxe._audit.data.checksoraxe._audit.checks, but I've had no luck.Edit 2
More info. I tried patching this package in my node_modules to return the rules and checks. When passing them to
axe-playwright, I get the following error:Edit 3
I tried switching from
axe-playwrightto@axe-core/playwright, and am still running into an issue: