Run axe-webdriverjs on sites with a limited Content Security Policy

[toolness]

Currently I'm using axe-webdriverjs to operate on third-party sites that I may not control. Some of them, such as github.com, don't seem to like axe-webdriverJS at all:

GitHub error output

Obtaining axe-core stats for https://github.com/18F/handbook.
/app/node_modules/selenium-webdriver/lib/promise.js:2634
        throw error;
        ^

JavascriptError: javascript error: axe is not defined
JavaScript stack:
ReferenceError: axe is not defined
    at eval (eval at executeAsyncScript (:439:5), :7:5)
    at eval (eval at executeAsyncScript (:439:5), :8:7)
    at eval (eval at executeAsyncScript (:439:5), :8:33)
    at executeAsyncScript (:439:26)
    at :455:29
    at callFunction (:347:33)
    at :357:23
    at :358:3
  (Session info: chrome=57.0.2987.110)
  (Driver info: chromedriver=2.28.455506 (18f6627e265f442aeec9b6661a49fe819aeeea1f),platform=Linux 4.4.43-boot2docker x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 39 milliseconds
Build info: version: '3.2.0', revision: '8c03df6', time: '2017-03-02 09:34:51 -0800'
System info: host: 'a2f7a9a3b0cb', ip: '172.21.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '4.4.43-boot2docker', java.version: '1.8.0_121'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, networkConnectionEnabled=false, chrome={chromedriverVersion=2.28.455506 (18f6627e265f442aeec9b6661a49fe819aeeea1f), userDataDir=/tmp/.org.chromium.Chromium.TH3pH4}, takesHeapSnapshot=true, pageLoadStrategy=normal, databaseEnabled=false, handlesAlerts=true, hasTouchScreen=false, version=57.0.2987.110, platform=LINUX, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true, unexpectedAlertBehaviour=}]
Session ID: 76fbe31eeed5528c2bb45bbc5e9d25d5
    at Object.checkLegacyResponse (/app/node_modules/selenium-webdriver/lib/error.js:517:15)
    at parseHttpResponse (/app/node_modules/selenium-webdriver/lib/http.js:516:11)
    at doSend.then.response (/app/node_modules/selenium-webdriver/lib/http.js:432:13)
    at process._tickCallback (internal/process/next_tick.js:109:7)
From: Task: WebDriver.executeScript()
    at mixin.schedule (/app/node_modules/selenium-webdriver/lib/webdriver.js:816:17)
    at mixin.executeAsyncScript (/app/node_modules/selenium-webdriver/lib/webdriver.js:900:17)
    at /app/node_modules/axe-webdriverjs/lib/index.js:108:5
    at /app/node_modules/axe-webdriverjs/lib/inject.js:63:4
    at ManagedPromise.invokeCallback_ (/app/node_modules/selenium-webdriver/lib/promise.js:1384:14)
    at TaskQueue.execute_ (/app/node_modules/selenium-webdriver/lib/promise.js:3092:14)
    at TaskQueue.executeNext_ (/app/node_modules/selenium-webdriver/lib/promise.js:3075:27)
    at asyncRun (/app/node_modules/selenium-webdriver/lib/promise.js:2935:27)
    at /app/node_modules/selenium-webdriver/lib/promise.js:676:7
    at process._tickCallback (internal/process/next_tick.js:109:7)

I suspect this may be due to Content Security Policy or somesuch, but that's besides the point right now--the main thing I'd like to accomplish is catching that error and e.g. move on to a different site. However, it seems that axe.analyze() doesn't have any way of passing an error back to the calling code, and instead ultimately causes the whole node process to crash with this uncaught exception.

Is there a way axe.analyze() can be modified to e.g. return a Promise that we can use to catch any errors or something?

[marcysutton]

I get a different error, but I can reproduce the problem:

VM35:329 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src assets-cdn.github.com". Either the 'unsafe-inline' keyword, a hash ('sha256-Sn4XRHw8EFtG1NWHkQGitqxPHVSvjehPowmi87qhWbQ='), or a nonce ('nonce-...') is required to enable inline execution.
(anonymous)
(anonymous)
(anonymous)
executeAsyncScript  @   VM35:329
(anonymous) @   VM35:345
callFunction    @   VM35:237
(anonymous) @   VM35:247
(anonymous) @   VM35:248

We should be able to bubble up an error by modifying axe-webdriverjs to use axe.run internally instead of axe.a11yCheck. But the browser just hangs in my testing, so we might need to put in a timeout of some kind so it will return the error after a while.

[toolness]

Cool, thanks for the quick response! Would you like me to start working on a PR to fix this?

[marcysutton]

Oh, sure! That would be awesome. I started changing it myself to validate it would work, that's how I discovered it would probably need a timeout. The relevant source is here: https://github.com/dequelabs/axe-webdriverjs/blob/master/lib/index.js#L113

To make use of axe.run, the calling function will have to pass another parameter to the callback. You can see the API docs for axe.run here: https://github.com/dequelabs/axe-core/blob/master/doc/API.md#api-name-axerun

[marcysutton]

This was addressed with #34 and b6fa546, we'll push an update to npm soon.

[toolness]

Awesome, thanks @marcysutton and @dylanb!