Written in Ruby, and assumes your HTML generating code is in Ruby. For GitHub, which I believe is still a Rails shop, this makes it easier for them to use, not so easy for us.
There is a command line option to test a live server (e.g. dev.mapzen.com), but the problem is that this means we have to deploy before we can test.
It may be possible to use the proofer in tests after HTML-generating, assuming we build test infrastructure in Ruby.
write-good
Written in Node, but provides a command-line interface. This makes it easier to incorporate into our build process and report failures via CircleCI.
Since this tests against content, ideally, we should be incorporating this on the content documentation repositories (at the cost of including some meta files in those repositories). If we ran this test in the generator repository, it's more inconvenient (impossible?) to report warnings and errors back to the original sources.
pa11y
Written in Node, but provides a command-line interface. This makes it easier to incorporate into our build process and report failures via CircleCI. However, the command-line tests a live server (e.g. dev.mapzen.com), and as noted above, testing a live server means we need to deploy before we can test at all.
If we want to run tests without deploying (so that we can deploy only if tests pass, which is better), this requires PhantomJS (basically, a pretend browser) in order to test how accessible things are. If we do this, we should plan for & take advantage of it to run an entire suite of front-end tests with the same system.
A brief summary of conclusions
Incorporate write-good in content repositories, rather than the generator repository. If we're okay with doing this and hooking up each content repository to CircleCI, the bonus feature is that each content repository can also run the generator and automatically deploy to dev when their tests pass.
It kind of sucks to have a bunch of things in one repository written in different languages, so we should see if pa11y and html-proofer have appropriate Python-based alternatives, and implement real tests.
Otherwise, we would compromise on having test infrastructure in one language even though the generator code is in Python. In this case I would use Node.js, use pa11y, and find a Node.js alternative to html-proofer. Why Node.js? Partially because I am more familiar with it, and partially because html-proofer seems designed to work best when the HTML generator is already Ruby, which, for us, it isn't.
These were recommended by GitHub staff and are used in GitHub's own help system.