search

mochajs / mocha

☕️ simple, flexible, fun javascript test framework for node.js & the browser
https://mochajs.org
MIT License
22.57k stars 3k forks source link

🐛 Bug: Mocha throws error if it is running a spec more than once #2706

Open crishushu opened 7 years ago

crishushu commented 7 years ago

taken from https://cdnjs.com/libraries/mocha

> mocha.run();

everything ok!

> mocha.run()

TypeError: Cannot read property 'call' of undefined
    at r (https://cdnjs.cloudflare.com/ajax/libs/mocha/3.2.0/mocha.min.js:2:8563)
    at r.run (https://cdnjs.cloudflare.com/ajax/libs/mocha/3.2.0/mocha.min.js:2:9635)
    at n (https://cdnjs.cloudflare.com/ajax/libs/mocha/3.2.0/mocha.min.js:2:13371)
    at https://cdnjs.cloudflare.com/ajax/libs/mocha/3.2.0/mocha.min.js:2:13710
    at i (https://cdnjs.cloudflare.com/ajax/libs/mocha/3.2.0/mocha.min.js:1:572)
dennisbaskin commented 7 years ago

Do you have an example of what necessitated multiple programatic runs of mocha?

tgirishkumar commented 7 years ago

I have the same issue: We have a test framework that runs tests written in mocha after doing some device setup. We earlier had a repeat feature where we called mocha.run n times if that feature was enabled.

We haven't tested this feature from long time but realized today that it is broken and we get same error as @crishushu mentioned.

Please confirm if Mocha supports repeating tests which can be controlled as an argument? I am aware of function within it/describe.

thanks

dennisbaskin commented 7 years ago

It is really hard to judge what this could be without a code example, including a setup. Out of curiosity have you looked https://github.com/mochajs/mocha/wiki/Using-mocha-programmatically ?

The wiki describes that the run method returns a Runnable, which might help you debug. For example: try running your second test after the first has completed and see if that helps:

Mocha.run().on('test end', runAgainFunc);

@tgirishkumar

Please confirm if Mocha supports repeating tests which can be controlled as an argument? I am aware of function within it/describe.

I am not sure what you mean here, could you elaborate?

tgirishkumar commented 7 years ago

I meant the retry feature: https://mochajs.org/#retry-tests , we are aware of it but wanted to retry/repeat tests without code changes.

thanks

tgirishkumar commented 7 years ago

Here is another example of someone raising the issue:

https://github.com/adamgruber/mochawesome/issues/79, Look at last comment:

Please note: this error happens in mochawesome but caused by this issue in Mocha inside, i tested by reproducing below issue and then remove mochawesome and then i find current issue.



Here's my code:

import * as path from 'path';
import * as fs from 'fs';
import * as Mocha from 'mocha';
import * as React from 'react';
import * as chokidar from 'chokidar';

let mocha: Mocha = new Mocha(
{
    reporter: 'mochawesome',
    reporterOptions:
    {
        reportDir: __dirname + '/../../public/mochawesome'
    }
});

let testsPath: string = __dirname + '/../../tests/';
let fileList: Array<string> = fs.readdirSync(testsPath).filter(file => file.split('.').pop() == 'js');
fileList.forEach (file =>
{
    console.log(file);
    mocha.addFile(path.join(testsPath, file));
});

function Invalidate()
{
    fileList.forEach (p =>
    {
        let fullPath = path.join(testsPath, p);
        p = path.resolve(fullPath);
        delete require.cache[require.resolve(p)];
    });
}

export default class Tests extends React.Component<{}, {}>
{
    componentWillMount()
    {
        //mocha.run();

        let watcher = chokidar.watch(testsPath, {persistent: true});
        watcher.on('change', path =>
        {
            console.log('change');
            Invalidate();
            mocha.run();
            //this.forceUpdate();
        });

        watcher.on('add', p =>
        {
            if (p.split('.').pop() == 'js')
            {
                console.log('add');
                Invalidate();
                mocha.addFile(p);
                mocha.run();
                //this.forceUpdate();
            }
        });
    }

    render()
    {

        return (
            <html>
                <body>
                    <iframe src='/mocha/mochawesome.html' width='100%' height='100%'/>
                </body>
            </html>
        );
    }
}

And my test code:

"use strict";
const chai_1 = require("chai");
describe('testing', function () {
    it('will equal 1', () => {
        chai_1.expect(1).to.equal(2);
    });
});

When the add or watch event is called more than once, it fails.```
elevow commented 7 years ago

Using 3.2.0 as well * edit: same behavior with 3.1.2 I am getting the same error and I am running mocha programatically already. My dept has a shared web component with multiple versions and we are testing that we did not break any version as prior versions of the component could be used by other teams. So we run the same tests on multiple versions of a component. As a side note, we use a SemVer check when new tests are added or need to be skipped.

The error seems to occur on the first line of code on the second time the file is run because I cannot get anything including logs. There are no problems or errors when I attempt to run only 1 version (so one run of the tests).

I thought maybe if I didn't use addFile multiple times that would work, but it did not make a difference.

Just a guess, I was wondering if the test is being cached like a require then would it need to be "unregistered" so it can be run again with new information?

Here is a brief output from my tests: global-nav-module_5.0.0 Component Tests Suite1 √ module should be added and render with option parameters true (12341ms)

1 passing (12s)

dennisbaskin commented 7 years ago

@tgirishkumar thank you for clarifying. I have never used mocha for React tests, nor have had a need for using retries on failed tests. I have generally setup tests to always run in one command (mocha <....>) from the command line. I would imagine it's easier to setup CI scripts that way, but have no way of comparing since I have never tried to do it the other way.

It is possible mocha is relying on something in the global state or loading setup multiple times. Hard to trace without a reproducible unit test / stack trace. Maybe with @elevow and @crishushu we can create a reproducible small unit test, track down the issue, and then submit a PR for the mocha team. They mentioned on their site that they are in dire need of help maintaining the project 😄

crishushu commented 7 years ago

Yesterday I was playing with different versions of mocha. AFAIK the latest release where running mocha with same suite/spec setting multiple times is supported, is 2.3.0. @dennisbaskin just out of curiosity is there any reason why the feature (yes, I consider it as one) has been disabled. I assume that it happened deliberately.

elevow commented 7 years ago

I was able to make a small version of the failure. Hopefully someone can duplicate it. one.test.js is just a it statement with console.log();

const Mocha = require('mocha');
const path = require('path');
const async = require('async');

const mocha = new Mocha({
  reporter: 'spec',
});
const counter = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];

const cwd = process.cwd();

mocha.addFile(path.join(cwd, 'one.test.js'));
async.eachLimit(counter, 1, (iCounter, eachCallback) => {
  mocha.run((failures) => {
    eachCallback();
  });
});

TypeError: Cannot read property 'call' of undefined

edit: I tried moving the addFile command inside and outside of the eachLimit and both give the failure

elevow commented 7 years ago

Hey after some testing the issue seems to be specific to async.eachLimit for me.

dennisbaskin commented 7 years ago

@crishushu I would not able to answer your question as to why. I am not directly involved on the contributing team, just trying to help triage issu.es since they need help.

@elevow I will try to play around with your example, thanks for setting that up.

qamate commented 7 years ago

Guys, any update on this?

tomdid commented 7 years ago

I have same problem. I just tried to set up mocha programmatically and have just one test is expect(1).toBe(1) . No fancy stuff, like react or enzyme is loaded.

I load one test file mocha.add('my.test.js')

then i run the tests.

mocha.run();
setTimeout(()=>{ mocha.run(); }, 5000);

First time it runs fine, after 5s it throws me an error. This is the full error stack TypeError: Cannot read property 'call' of undefined at callFn ([__dirname]\node_modules\mocha\lib\runnable.js:348:20) at Test.Runnable.run ([__dirname]\node_modules\mocha\lib\runnable.js:340:7) at Runner.runTest ([__dirname]\node_modules\mocha\lib\runner.js:443:10) at [__dirname]\node_modules\mocha\lib\runner.js:549:12 at next ([__dirname]\node_modules\mocha\lib\runner.js:361:14) at [__dirname]\node_modules\mocha\lib\runner.js:371:7 at next ([__dirname]\node_modules\mocha\lib\runner.js:295:14) at Immediate.<anonymous> ([__dirname]\node_modules\mocha\lib\runner.js:339:5) at runCallback (timers.js:800:20) at tryOnImmediate (timers.js:762:5)

qamate commented 7 years ago

I think you should have a look at grunt-loop-mocha.

I use it for my Selenium WebDriver tests.

https://github.com/grawk/grunt-loop-mocha

My understanding is that mocha is a unit testing framework and so it creates only one instance/process of your tests. So, when you try to execute the same test, it does not execute before and after statements.

Grunt-loop-mocha on other hand helps you create the new instance for each specification. So, it will even execute before and after statements with the different configuration. I guess that's what you are looking for.

karneaud commented 6 years ago

+1

mocha --compilers js:babel-register --allow-uncaught --full-trace --no-warnings --recursive tests/e2e-tests

got this error

Frontend Testing "before all" hook: open browser: TypeError: Cannot read property 'call' of undefined at callFn (/usr/src/app/node_modules/mocha/lib/runnable.js:348:20) at Hook.Runnable.run (/usr/src/app/node_modules/mocha/lib/runnable.js:340:7) at next (/usr/src/app/node_modules/mocha/lib/runner.js:309:10) at Immediate. (/usr/src/app/node_modules/mocha/lib/runner.js:339:5) at runCallback (timers.js:781:20) at tryOnImmediate (timers.js:743:5) at processImmediate [as _immediateCallback] (timers.js:714:5)

Frontend Testing "after all" hook: quit browser: TypeError: Cannot read property 'call' of undefined at callFn (/usr/src/app/node_modules/mocha/lib/runnable.js:348:20) at Hook.Runnable.run (/usr/src/app/node_modules/mocha/lib/runnable.js:340:7) at next (/usr/src/app/node_modules/mocha/lib/runner.js:309:10) at Immediate. (/usr/src/app/node_modules/mocha/lib/runner.js:339:5) at runCallback (timers.js:781:20) at tryOnImmediate (timers.js:743:5) at processImmediate [as _immediateCallback] (timers.js:714:5)

my test

import config from 'config';
import chai from 'chai';
import chaiAsPromised from 'chai-as-promised';
import test from 'selenium-webdriver/testing';
import webdriver from 'selenium-webdriver';
import proxy from 'selenium-webdriver/proxy';
import chrome from 'selenium-webdriver/chrome';
// Helper objects for performing actions.
import { WebDriverManager, WebDriverHelper as helper } from 'wp-e2e-webdriver';

// We're going to use the ShopPage and CartPage objects for this tutorial.
import { ShopPage, CartPage } from 'wc-e2e-page-objects';

chai.use( chaiAsPromised );
const assert = chai.assert;
const chromedriver = require('chromedriver');
const chromeCapabilities = webdriver.Capabilities.chrome();
chromeCapabilities.set('chromeOptions', {args: ['--headless --no-sandbox user-agent=Mozilla/5.0 (wp-e2e-tests) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36' ]});
const pref = new webdriver.logging.Preferences();
pref.setLevel( 'browser', webdriver.logging.Level.SEVERE );

let manager;
let driver;

test.describe( 'Frontend Testing', function() {

    // Set up the driver and manager before testing starts.
    test.before( 'open browser', function() {
        driver = new webdriver.Builder()
          .forBrowser('chrome')
          .usingServer( config.get("url") )
          .withCapabilities(chromeCapabilities)
            .setLoggingPrefs(pref)
            .setProxy(proxy.direct())
          .build();
        driver.getSession().then( function( sessionid ) {
            driver.allPassed = true;
            driver.sessionID = sessionid.id_;
        } );

        helper.clearCookiesAndDeleteLocalStorage( driver );
    } );
    // Tests will go here.
    test.it( 'Has shop page', () => {

            // Create a new Shop page object.
            const shopPage = new ShopPage( driver, { url: config.get('url').concat( '/shop' ) } );

            assert.instanceOf(shopPage, ShopPage, 'shopPage instance of ShopPage');

        } );
    // Close the browser after finished testing.
    test.after( 'quit browser', () => {
        driver.sleep( config.get('startBrowserTimeoutMs') ).then( () => {
            return driver.quit();
        } );
    } );

} );

I'm running in a docker environment with

node 8.5.0 npm 5.4.2 mocha 3.5.3

super9user commented 6 years ago

This worked for me, if anyone is still looking for a solution

const purge = function (allFiles) {
  allFiles.forEach(file => {
    delete require.cache[file];
  });
};
// use this function when rerunning tests
const rerun = function () {
  purge(allFiles);
  mocha.suite = mocha.suite.clone();
  mocha.suite.ctx = new Mocha.Context();
  mocha.ui("bdd");
  mocha.files = allFiles;
  mocha.run()
};
kraenhansen commented 6 years ago

I am seeing this issue too using Mocha 5.2.0 - here is my stacktrace:

TypeError: Cannot read property 'call' of undefined
  at callFn (/.../node_modules/mocha/lib/runnable.js:372:21)
  at Test.Runnable.run (/.../node_modules/mocha/lib/runnable.js:364:7)
  at Runner.runTest (/.../node_modules/mocha/lib/runner.js:455:10)
  at /.../node_modules/mocha/lib/runner.js:573:12
  at next (/.../node_modules/mocha/lib/runner.js:369:14)
  at /.../node_modules/mocha/lib/runner.js:379:7
  at next (/.../node_modules/mocha/lib/runner.js:303:14)
  at Immediate.<anonymous> (/...node_modules/mocha/lib/runner.js:347:5)
  at runCallback (timers.js:794:20)
  at tryOnImmediate (timers.js:752:5)
  at processImmediate [as _immediateCallback] (timers.js:729:5)

I am writing a library where I want to compare the output to console of a reporter based with and without my library .. which is why I need to be able to re-run the same test twice, programatically.

When inspecting the Mocha.Test objects I see that they are missing a fn property the second time they run, which probably is why runnable.js:372 is calling an undefined function.

It looks like a Mocha.Suite is cleaning up by deleting these fn properties to avoid memory leaks https://github.com/mochajs/mocha/blob/master/lib/runner.js#L817-L819.

I ultimately gave up and just reinstantiated the mocha instance between runs - remembering to clear node.js require cache (delete require.cache[testPath]) between runs.

RyanLiuF commented 5 years ago

I clean the require cache up by using the clear-require component, and it works every time, the code as follows: const clearRequire = require('clear-require'); clearRequire.match(new RegExp('mocha'));//must be here clearRequire.match(new RegExp('your-test-js-file'));//must be here more details : const clearRequire = require('clear-require'); function testMochaClick() { clearRequire.match(new RegExp('mocha')); clearRequire.match(new RegExp('commtest')); let Mocha = require('mocha'); let mocha = new Mocha({ reporter: 'mochawesome', reporterOptions: { reportDir: './resources/app/report', } }) mocha.addFile('./resources/app/case/commtest.js'); mocha.run(); } https://mochajs.org/api/mocha#loadFiles

plroebuck commented 5 years ago

The current codebase needs mocha to be reinstantiated between runs due to internal implementation issues.

JoshuaKGoldberg commented 9 months ago

Related: #1938

JoshuaKGoldberg commented 8 months ago

Looking back at this issue and #1938: fixing it would be a pretty significant internal refactor. I think we can accept PRs for it, but note that per #5027 it'll be a while before we can review them.