intermittent failure

[xavierhaurie]

Issue description or question

Test code fails, and then is made to pass by simply adding a console.log() statement. Restart the runner and it fails again. Remove the console.log() and it passes... etc.

Wallaby.js configuration file

/*
 * Copyright (c) 2015-2016, President and Fellows of Harvard College
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *
 * 1. Redistributions of source code must retain the above copyright
 * notice, this list of conditions and the following disclaimer.
 *
 * 2. Redistributions in binary form must reproduce the above copyright
 * notice, this list of conditions and the following disclaimer in the
 * documentation and/or other materials provided with the distribution.
 *
 * 3. The name of the author may not be used to endorse or promote products
 * derived from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED
 * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO
 * EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
 * TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */
module.exports = function (wallaby) {
    return {
        "files": [
            {pattern: 'node_modules/sinon/pkg/sinon.js', instrument: false},

            'src/main/webapp/js/lib/json2.js',
            'src/main/webapp/js/lib/jquery.min.js',
            'src/main/webapp/js/lib/jquery-ui.min.js',
            'src/main/webapp/js/lib/maskedInputPlugin.js',

            'src/main/webapp/js/lib/date.format.js',
            'src/main/webapp/js/lib/jquery.blockUI.js',
            'src/main/webapp/js/util.js',
            'src/main/webapp/js/dto.js',
            'src/main/webapp/js/breadcrumbs.js',
            'src/main/webapp/js/global2.js',

            'src/main/webapp/js/home_screen.js',
            'src/main/webapp/js/feedback_screen.js',
            'src/main/webapp/js/booked_resources_table.js',
            'src/main/webapp/js/home_appointment.js',
            'src/main/webapp/js/appointment_screen.js',
            'src/main/webapp/js/appointment_module.js',
            'src/main/webapp/js/appt_calendar.js',
            // needed by app_calendar.html page
            'src/main/webapp/js/appt_calendar_page.js',
            'src/main/webapp/js/home_study.js',
            'src/main/webapp/js/study_screen.js',
            'src/main/webapp/js/study_form.js',
            'src/main/webapp/js/study_module.js',
            'src/main/webapp/js/subjectGrid.js',

            'src/main/webapp/js/lib/fullcalendar-scheduler/lib/moment.min.js',
            'src/main/webapp/js/lib/fullcalendar-scheduler/lib/fullcalendar.min.js',

            'src/main/webapp/js/lib/jquery-ui-sliderAccess.js',
            'src/main/webapp/js/lib/jquery-ui-timepicker-addon.js',
            'src/main/webapp/js/lib/jsrender.js',

            'src/main/webapp/**/*.css',

            'src/main/webapp/**/*.html',

            {pattern: 'node_modules/jasmine-jquery/lib/jasmine-jquery.js', instrument: false},
            {pattern: 'src/test/jasmine/helpers/**/*.js', instrument: false}

            //'node_modules/sinon/lib/sinon.js',
        ],

        "tests": [
            'src/test/jasmine/**/*Spec.js'
        ],

        "testFrameworks": ['jasmine', 'sinon'],

        "debug": true
    };

};

Code editor or IDE name and version

IntelliJ IDEA v15

OS name and version

OSX

[ArtemGovorov]

We have seen issues like this before and in all cases the reason was a dependency between tests. Unlike karma, that always runs all tests together, wallaby runs tests in parallel, so neither the execution order, nor the set of test files that will be executed together can't be guaranteed.

From what you've described, some of your tests may fail because they somehow depend on other tests. When you edit a specific files, wallaby runs just that file, so nothing influences its tests and it passes.

I'd start with setting both workers count to one to see if the issue is still there and remove the parallelisation factor before it's sorted out.

Then I'd try to modify tests setting to only include the spec where the fantom error shows up. Running the config with just that spec should clear the error. Then I'd go ahead and include some other specs, while restarting wallaby to see if the issue shows up again (like bisecting by the other way around). The idea is basically to find out what other spec is influencing this one.

If you don't see any issues with your test dependencies, please create a sample GH project where the issue can be reproduced, I'm happy to have a look.

[ArtemGovorov]

Closing the issue for now, please let me know if it's still valid and you can share a sample project so I may have a look.