Closed dmundra closed 2 years ago
Hi @dmundra,
Could you please provide a public URL that we can test? It's impossible for us to debug the issue otherwise.
I've noticed you're using Node 17, could you please test it again using an LTS version of Node, as those are the only ones that we support?
If I'm reading your example correctly and the URL that you're testing has the same content as https://accessibility.civicactions.com/guide/glossary, this is working correctly with 3.0.0:
Broadly speaking, the error that you show there is a puppeteer error, so there's very little that we can do about it besides updating puppeteer to a latest version. I've seen similar errors in the past with puppeteer choking on servers that are configured in a specific way. If pa11y runs successfully on a public facing URL, but fails to run on a localhost one, this may be the same problem.
Hi @josebolos,
I am testing locally and in GitHub actions for the site you mentioned, https://github.com/CivicActions/accessibility/pull/580#issuecomment-981121523
I got the same error when trying it against the live sitemap.xml
./node_modules/pa11y-ci/bin/pa11y-ci.js --sitemap https://accessibility.civicactions.com/sitemap.xml
Running Pa11y on 119 URLs:
> https://accessibility.civicactions.com/guide/events - 2 errors
> https://accessibility.civicactions.com/guide/glossary - Failed to run
/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71
this._callbacks.set(id, { resolve, reject, error: new Error(), method });
^
Error: Protocol error (Target.disposeBrowserContext): Failed to find context with id F6EC73A10C23F6F16F75D9DCE662CC99
at /Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71:63
at new Promise (<anonymous>)
at Connection.send (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:70:16)
at Browser._disposeContext (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:168:32)
at BrowserContext.close (/Users/dmundra/workspaces/accessibility/node_modules/puppeteer/lib/cjs/puppeteer/common/Browser.js:523:29)
at testRunner (/Users/dmundra/workspaces/accessibility/node_modules/pa11y-ci/lib/pa11y-ci.js:135:21)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
Node.js v17.0.1
I am also running with axe but I get the error even when not running with axe. I will try different version of node as well.
I did find this puppeteer issue https://github.com/puppeteer/puppeteer/issues/6258 if it helps. Maybe some of those versions needs to be changed for pa11y-ci.
Same error with node LTS version v16.13.0
I couldn't re-create it locally by running
npx pa11y-ci --sitemap https://accessibility.civicactions.com/sitemap.xml --sitemap-exclude "/*.pdf"
If possible, could you share your pa11y-ci
config file? And also the output of npx pa11y --environment
If possible, could you share your
pa11y-ci
config file? And also the output ofnpx pa11y --environment
This is @dmundra's config file for that site:
{
"defaults": {
"concurrency": 4,
"standard": "WCAG2AA",
"runners": ["axe"]
}
}
He's also using ubuntu-latest
in his test action, so it would be similar to my environment:
System:
OS: Linux 5.11 Ubuntu 20.04.3 LTS (Focal Fossa)
CPU: (2) x64 Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Memory: 4.13 GB / 6.79 GB
Shell: 5.0.17 - /bin/bash
Binaries:
Node: 16.13.0 - /opt/hostedtoolcache/node/16.13.0/x64/bin/node
Yarn: 1.22.17 - /usr/local/bin/yarn
npm: 8.1.0 - /opt/hostedtoolcache/node/16.13.0/x64/bin/npm
pa11y: 6.1.1
I'm testing on a local GitHub Actions --detached
site so I don't blow out my Google Analytics every time this runs; so, please don't run on my production sites to test (can't speak for @dmundra).
For me, this runs fine on my local Windows 10 machine. On a side note, I'm running html-proofer
on this same --detached
site on ubuntu-latest
without problems, so I don't think it's the host.
I suspect this has to do with concurrency. I changed my config file to "concurrency": 1
and it ran all the way through with Exit Code 2; however, it's producing false negatives that do not show up when I run locally.
Changing config "runners": ["axe"]
to "runners": ["htmlcs"]
solves my false negative problem.
In summary, there are likely two things that don't work in this scenario:
The workaround is to change configurations in .pa11yci
to:
{
"defaults": {
"concurrency": 1,
"standard": "WCAG2AA",
"runners": ["htmlcs"]
}
}
Same error with node LTS version v16.13.0
Same issue on CircleCI's cimg/ruby:3.0-browsers
latest image with node v16.13.0
/home/circleci/project/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:71
this._callbacks.set(id, { resolve, reject, error: new Error(), method });
.pa11yci
:
{
"defaults": {
"concurrency": 4
}
}
Thanks @DaveSkender.
I'm testing on a local GitHub Actions --detached site so I don't blow out my Google Analytics every time this runs; so, please don't run on my production sites to test (can't speak for @dmundra).
I am doing the same in GitHub Actions, i.e. running against a local detached Jekyll site. The relevant files:
Confirmed that reducing concurrency to 1 allows the process to run to completion.
@joeyciechanowicz @josebolos I believe it's the same core issues I came across tonight working on a reporter, and in longer running reporter.afterAll
functions (with a couple of reporters, with layers of async) it fails with:
E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:71
this._callbacks.set(id, { resolve, reject, error: new Error(), method });
Error: Protocol error (Target.disposeBrowserContext): Target closed.
at E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:71:63
at new Promise (<anonymous>)
at Connection.send (E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Connection.js:70:16)
at Browser._disposeContext (E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Browser.js:168:32)
at BrowserContext.close (E:\project\node_modules\puppeteer\lib\cjs\puppeteer\common\Browser.js:523:29)
at testRunner (E:\project\node_modules\pa11y-ci\lib\pa11y-ci.js:135:21)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
It looks to be a missing await
at the specified line in the call stack - browserContext.close
is an async function. Running with "useIncognitoBrowserContext": false
seems to be a workaround since it avoids the line, but adding the await looks like it solved the core issue (at least in my case).
I have a PR if you want it, but it is just the one line.
After #128, I updated to 3.0.0 but I am getting a puppeteer error. Am I missing something?