garris / BackstopJS

Catch CSS curve balls.
http://backstopjs.org
MIT License
6.72k stars 602 forks source link

Puppeteer engine scripts don't seem to be running #881

Open aliu145 opened 6 years ago

aliu145 commented 6 years ago

After changing the engine on our configuration file from chromy to puppeteer, it doesn't seem like the engine_scripts files are running i.e. onReady and clickAndHoverHelper (the console.log in these files are never printed, and clickSelector, hoverSelector etc don't result in the expected actions). Any ideas why this may be?

const backstop = require('backstopjs');
const path = require('path');
const fs = require('fs');
const argv = require('yargs').argv;

let backstopMode = 'test';
let allscenarios = [];
let dns = argv.e;
// if component is specified,retrieves scenarios from the component json file
if (argv.c != null) {
    allscenarios = require(path.resolve(__dirname, `./tests/${argv.c}.json`)).scenarios;
} else {
    // if no component is specified,build scenarios from all components from scenarios folder
    const files = fs.readdirSync(path.resolve(__dirname, 'tests'));
    files.forEach((file) => {
        const iconScenarios = require(path.resolve(__dirname, `./tests/${file}`)).scenarios;
        allscenarios = allscenarios.concat(iconScenarios);
    });
}
// set backstop command to 'test' if user inputs reference option
if (argv.r == 'reference') {
    backstopMode = 'reference';
}

// Use default env if env option is not specified
const default_environment = 'http://localhost:4001';
if (!argv.e) {
    dns = default_environment;
}

// Replace URL in all scenarios
const tempScenarios = JSON.stringify(allscenarios).replace(/\$dns/g, dns);
allscenarios = JSON.parse(tempScenarios);

// building backstop config file for headless chrome
backstop(backstopMode, {
    config: {
        id: 'test',
        viewports: [
            {
                name: 'desktop',
                width: 1920,
                height: 2500,
            },
        ],
        scenarios: allscenarios,
        paths: {
            engine_scripts: path.resolve(__dirname, 'utils'),
            bitmaps_reference: path.resolve(__dirname, 'test-output/chrome/expected_result'),
            bitmaps_test: path.resolve(__dirname, 'test-output/chrome/actual_result'),
            html_report: path.resolve(__dirname, 'test-output/chrome/html_report'),
            ci_report: path.resolve(__dirname, 'test-output/chrome/ci_report'),
        },
        engine: 'puppeteer',
        engineOptions: {
            args: ['--no-sandbox'],
        },
        report: ['browser', 'CI'],
        debug: false,
        delay: 500,
        misMatchThreshold: 0.1,
    },
    asyncCaptureLimit: 2,
    asyncCompareLimit: 25,
}).catch((e) => {
    console.log(e);
    process.exit(1);
});
garris commented 6 years ago

Please post your config...

aliu145 commented 6 years ago

Config posted, thanks!

garris commented 6 years ago

Can you confirm your script files are actually found — also if you change to puppeteer (which is recommended) you need to use puppeteer compatible scripts. (Check the puppeteer script directory for templates)

aliu145 commented 6 years ago

Hi, yes. Sorry to give more information, my script files are running, and my own custom onReady scripts run as well, but the params used in clickAndHoverHelper do not work, and also I see in onReady.js it has this code:

module.exports = async (page, scenario, vp) => {
  console.log('SCENARIO > ' + scenario.label);
  await require('./clickAndHoverHelper')(page, scenario);

but I'm not seeing that logging in my console (i.e. the SCENARIO > scenario.label part)

garris commented 6 years ago

I think you just need to look at each file in your scripting flow -- add break points and console logs until you find the point where your the ./clickAndHoverHelper is not getting called. You can use a clean backstop implementation (via backstop init) on your machine somewhere as a template. Good luck.