Error running backstop test with docker container

[dodozhang21]

Here's my configuration file

{
  "id": "test",
  "viewports": [
    {
      "label": "small",
      "width": 320,
      "height": 480
    },
    {
      "label": "medium",
      "width": 568,
      "height": 1024
    },
    {
      "label": "large",
      "width": 1024,
      "height": 768
    },
    {
      "label": "xlarge",
      "width": 1291,
      "height": 768
    }
  ],
  "scenarios": [
    {
      "label": "Family Circle Story",
      "url": "http://www.familycircle.com/family-fun/10-books-to-read-now/",
      "hideSelectors": [
        "#ad",
        "#mobileHoverBannerWrapper", 
        "#header-banner-container",
        "#sidebar-second"
      ], 
      "removeSelectors": [
        "#block-outbrain-integration-outbrain",
        "#monetate_lightbox",
        "#block-medianet-medianet"
      ],
      "selectorExpansion": true,
      "selectors": [],
      "readyEvent": null,
      "delay": 1000,
      "misMatchThreshold" : 10,
      "requireSameDimensions" : true
    }
  ],
  "paths": {
    "bitmaps_reference": "backstop_data/bitmaps_reference",
    "bitmaps_test": "backstop_data/bitmaps_test",
    "engine_scripts": "backstop_data/engine_scripts",
    "html_report": "backstop_data/html_report",
    "ci_report": "backstop_data/ci_report"
  },
  "engineFlags": [],
  "engine": "chrome",
  "report": ["browser"],
  "debug": true,
  "debugWindow": false
}

and after trying to run this using the container via docker run --rm -v $(pwd):/src docksal/backstopjs test, I get the following error

CasperJS: [info] [phantom] Step anonymous 8/28: done in 5491ms.
CasperJS:  [info] [phantom] Step anonymous 9/28 http://www.familycircle.com/family-fun/10-books-to-read-now/ (HTTP 200)
CasperJS:  [info] [phantom] Step anonymous 9/28: done in 5508ms.
CasperJS:  [info] [phantom] Step anonymous 10/28 http://www.familycircle.com/family-fun/10-books-to-read-now/ (HTTP 200)
CasperJS:  [debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=true
CasperJS:  [debug] [phantom] url changed to "about:blank"

Testing script failed with code: 1

An unexpected error occured. You may want to try setting the debug option to `true` in your config file.
COMMAND | Command `test` ended with an error
COMMAND | Error: An unexpected error occured. You may want to try setting the debug option to `true` in your config file.
              at ChildProcess.<anonymous> (/usr/lib/node_modules/backstopjs/core/util/createBitmaps.js:70:18)
              at emitTwo (events.js:106:13)
              at ChildProcess.emit (events.js:191:7)
              at maybeClose (internal/child_process.js:877:16)
              at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)

When run the tests on my local machine (mac OS) it runs fine. Also I don't understand why inside the container it uses casperjs instead of chrome.

If you have any pointers, it will be appreciated.

Thanks

[garris]

My guess is that you have 2 configs? Is one local to your mac and the other local to your container?

[dodozhang21]

I mounted my pwd to the container, shouldn't it be using the same file? My config file is at $(pwd)/backstop.json

And as you see in the log, it's loading the correct url CasperJS: [info] [phantom] Step anonymous 9/28 http://www.familycircle.com/family-fun/10-books-to-read-now/ (HTTP 200)

I copied the command directly from https://github.com/docksal/backstopjs/

[garris]

Ah ha -- That docker image is totally bootleg! 😄

You may want to request the maintainer update his container. It's probably running an old version.

[dodozhang21]

If I were to build one from scratch is my understanding correct that https://gist.github.com/dodozhang21/11a91452d9a1afd6133775d96e2b8266 should work for me?

BTW I'm running backstop via npm run backstop:test

package.json

{
  "name": "mdp.node.tests",
  "version": "0.0.0",
  "scripts": {
    "test": "npm run backstop:test && npm run selenium:test",
    "selenium:test": "mocha $(find tests -name *Test.js ! -name _*) --timeout 25000 -R spec",
    "selenium:test:docker": "env SELENIUM_REMOTE_URL=http://chrome:4444/wd/hub npm run selenium:test",
    "backstop:reference": "backstop reference",
    "backstop:test": "backstop test",
    "backstop:approve": "backstop approve"
  },
  "devDependencies": {
    "backstopjs": "^3.0.20",
    "chai": "^4.1.0",
    "chai-as-promised": "^7.1.1",
    "chromedriver": "^2.31.0",
    "mocha": "^3.5.0",
    "selenium-webdriver": "^3.5.0"
  }
}

When I tried to run those tests inside my custom container, it times out :(

COMMAND | Executing core for `test`
createBitmaps | Selected 1 of 1 scenarios.
Starting Chromy: port:9222 --disable-gpu,--force-device-scale-factor=1,--disable-infobars=true,--window-size=320,480
Starting Chromy: port:9223 --disable-gpu,--force-device-scale-factor=1,--disable-infobars=true,--window-size=568,1024
Starting Chromy: port:9224 --disable-gpu,--force-device-scale-factor=1,--disable-infobars=true,--window-size=1024,768
Starting Chromy: port:9225 --disable-gpu,--force-device-scale-factor=1,--disable-infobars=true,--window-size=1291,768
COMMAND | Command `test` ended with an error after [31.086s]
COMMAND | BackstopException: Family Circle Story on undefined: GotoTimeoutError: goto() timeout

[garris]

@dodozhang21 Unfortunately I don't have much docker experience -- But I know other contributors are using backstop in this way. If you can solve your problem please post back here -- I am sure there are others who are trying to figure this one out.

@swese44 -- did you ever post your docker/backstop project?

@mirzazeyrek -- maybe you know someone using a container to run a backstopjs app?

[swese44]

yes! we run visual diffs as part of our full test run. locally this needs to happen in docker so when Travis CI runs get triggered by PRs and merges they are running in the same environment. note that we've had a few issues keeping Docker and Travis environments identical, and we might consider Screener.io as a remote visual diff runner rather than Docker + Backstop.

Here's the project: https://github.com/Microsoft/YamUI/

Interesting bits are our npm scripts, Dockerfile, docker-compose.yml for setting up the container to play nice with our file structure, yarn-docker for piping yarn commands under alias y into the Docker container, this terrible script to collect our list of Storybook stories, and our backstop config.

[swese44]

my team develops on Macbooks (yes we work at Microsoft) so your mileage may vary if you're on Windows, haven't tested there

[garris]

@swese44 Fantastic -- thank you for sharing all this work!

Fun fact: we both work for Microsoft 😄. Nice to meet you!

[dodozhang21]

@swese44 I'm not sure how your project helps me. Does your docker container only pull down node and yarn? Doesn't backstop need chromy to run? My docker container contains node, chrome and when I run backstop test under it, it just hangs. What exactly do you do?

Basically I just need a docker container that will successfully run backstop test via node/package.json

[swese44]

We're not using Chrome/Chromy, just PhantomJS. looks like this is newly-supported, we're still on BackstopJS 2.x. If you want to use PhantomJS instead then our config setup should help you, but keep in mind it's basically an old version of Firefox so you'll need autoprefixer if you need to support flexbox or other modern CSS rules.

[lmakarov]

@dodozhang21 https://github.com/docksal/backstopjs is built with BackstopJS v2.6.9, PhantomJS, SlimerJS and Firefox ESR. It does not support the new 3.x version and the added chrome driver (yet). You will have to stick with BackstopJS 2.x and phantomjs/slimerjs engines if you want to use that container image.

I'm looking into upgrading the image to use Backstop3 and support chrome engine, but it does not seem to be a simple version bump upgrade.

[lmakarov]

docksal/backstopjs has been upgraded to BackstopJS v3 with Headless Chrome support.

@dodozhang21 do a docker pull docksal/backstopjs to get the latest image version.

I wrote an article on how to use BackstopJS in a Docker container: https://blog.docksal.io/visual-regression-testing-with-backstopjs-in-a-docker-container-dfd1b9ae8582

[dodozhang21]

@lmakarov Thank you so much! I will definitely try it out!