Support modern browsers, not just PhantomJS

bergie commented 8 years ago

PhantomJS 2, uses WebKit 538.1, which was released in November 2013. Significant improvements have been done in browsers in these 2+ years. Both in terms of performance, new HTML apis (not so relevant for us) and new language support (ES6/ES2015 etc). Because Qt Project deprecated the QtWebkit they use in favor of Blink-based QtWebEngine, a more modern version would only come with (yet unplanned) PhantomJS 3 which would rewrite this...

The challenge is to have a headless setup suitable for running in production.

Alternatives

Chromium (Blink). Communicate through WebDriver/Selenium, or run-headless-chrome
Firefox (Gecko). Communicate through WebDriver/Selenium, or SlimerJS
Servo. Communicate/setup through?? Fast but highly experiemental webengine, part of Mozilla's Rust project

Headless approaches

Get xfvb-run (virtual X11) to work on Heroku. Possible but not easy
Use some virtual/headless Wayland compositor instead of X11. Weston headless. But neither Firefox nor Chromium has good mainline support yet.
Use the native headless support in Servo
Maybe try Ozone headless support in Chromium Content Shell. Looks not very mature
Run Docker images with virtual X11. On Amazon AWS ECS or similar, instead of Heroku.
Build our own headless browser shell, for instance based on QtWebEngine or Chromium Embedding framework
Native headless backend for Firefox or for Chromium. Large undertaking

Related to #6

jonnor commented 8 years ago

Chromium headless looks to have been worked on actively the last couple of months. Instructions on how to build here: https://chromium.googlesource.com/chromium/src/+/master/headless/README.md

It will require a custom build Chromium for foreseeable future, since graphics backend is compile time choice and Linux builds use X11 or Wayland probably.

jonnor commented 7 years ago

Looks like QtWebkit has been resurrected, with ES2015 etc coming to PhantomJS 2.5: https://groups.google.com/forum/m/#!topic/phantomjs/AefOuwkgBh0

bergie commented 7 years ago

Chrome headless will be shipping in 59 for all platforms:

https://www.chromestatus.com/features/5678767817097216

jonnor commented 7 years ago

And the PhantomJS maintainer stepped down, https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuNE

jonnor commented 7 years ago

Headless API for Chrome, by official Chrome team and using a standard binary build of Chrome: https://github.com/GoogleChrome/puppeteer

jonnor commented 7 years ago

Some work is still going on in PhantomJS, moving towards a 2.5 release: https://github.com/ariya/phantomjs/issues/14458

flowhub / jsjob

Support modern browsers, not just PhantomJS #7