YahooArchive / arrow

FE Test framework designed to promote TDD
http://yahoo.github.io/arrow/arrow_intro.html
BSD 3-Clause "New" or "Revised" License
55 stars 59 forks source link

Test over 30 seconds breaks (node.js, engine=mocha) #157

Closed maxiaohao closed 10 years ago

maxiaohao commented 10 years ago

For a test MyTest.js with its 'it' block takes longer than 30 seconds:

var expect = require("chai").expect,
waitSeconds = 40; 

describe('My Test -> ', function() {
    it('should pass after ' + waitSeconds + ' seconds', function(done) {
        // explicitly set a longer timeout to make sure test can finish
        this.timeout((waitSeconds + 1) * 1000);
        // takes more than 30 seconds
        setTimeout(function() {
            expect(1).to.equal(1);
            done();
        },  
        waitSeconds * 1000);
    }); 
});

The test passes if run with mocha: mocha MyTest.js -R spec

My Test -> 
    ✓ should pass after 40 seconds (40006ms)

  1 test complete (40 seconds)

But it breaks when run with arrow: arrow MyTest.js --engine=mocha

 My Test -> 
    ◦ should pass after 40 seconds: [2013-12-29 21:12:36.375] [INFO] runNodejsTest - Waiting for the test report
[2013-12-29 21:12:36.877] [INFO] runNodejsTest - Waiting for the test report
[2013-12-29 21:12:37.377] [INFO] runNodejsTest - Waiting for the test report
...
[2013-12-29 21:13:05.922] [INFO] runNodejsTest - Waiting for the test report
    1) should pass after 40 seconds
{
  "title": "should pass after 40 seconds",
  "async": 1,
  "sync": false,
  "timedOut": false,
  "pending": false,
  "type": "test",
  "parent": "#<Suite>",
  "ctx": "#<Context>",
  "timer": {
    "ontimeout": null
  },
  "state": "failed",
  "err": {
    "uncaught": true
  }
}

  ✖ 1 of 1 test failed:

  1) My Test ->  should pass after 40 seconds:
     TypeError: Cannot read property 'length' of undefined
      at onReportReady (/home/kevin/dev/tlswe/aws-mock/node_modules/yahoo-arrow/nodejs/node.js:61:63)
      at null.<anonymous> (/home/kevin/dev/tlswe/aws-mock/node_modules/yahoo-arrow/nodejs/node.js:94:17)
      at wrapper [as _onTimeout] (timers.js:252:14)
      at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)

Environments: arrow: v0.0.88 mocha: 1.7.4 (as dependency in arrow) node.js: v0.10.22

Is that a bug, or is there any switch or solution for enlarging arrows' default timeout (especially by using engine=mocha)?

Thanks very much!

maxiaohao commented 10 years ago

Pull request #158 can fix the mistaken code, but the test still breaks after 30-second timeout, only with some "more friendly" messages returned:

[2014-01-04 00:41:49.154] [INFO] runNodejsTest - Waiting for the test report
...
[2014-01-04 00:42:17.689] [INFO] runNodejsTest - Waiting for the test report
[2014-01-04 00:42:18.189] [INFO] runNodejsTest - Waiting for the test report
[2014-01-04 00:42:18.195] [INFO] TestExecutor - Time taken to run - undefined-Default is::31.63
[2014-01-04 00:42:18.199] [INFO] ReportManager - No TestReport found, but passing the test because 'hasTest' was passed as 'false'

It seems that probably Arrow's reporting thread could wait no longer than 30 seconds. Does anyone know the right way to enlarge timeout? Or any good resolution?

Thanks!

maxiaohao commented 10 years ago

Fixed by enlarging the timeout with command line option --testTimeOut. E.g. arrow MyTest.js --engine=mocha --testTimeOut=120000 can take up to 2 minutes. Thank Pranav!

Closing this issue.