Tests fail on clean clone on OSX due to timing issues

[justrhysism]

Node v0.12.7

git clone {url} npm i npm run test

Output (reduced for brevity):

...successful tests... glob plugin ✓ globs a wildcard ✓ globs a wildcard and forwards initial input events ✓ globs a wildcard using the basePath option ✓ globs two wildcards ✓ detects changes to two files matching globbed pattern (765ms)

/Users/rhys.lloyd/Projects/sigh/node_modules/sigh-core/node_modules/baconjs/dist/Bacon.js:361 after(); ^ AssertionError: expected [ Array(2) ] to deeply equal [ Array(1) ] at /Users/rhys.lloyd/Projects/sigh/src/test/plugin/glob.spec.js:143:28 at /Users/rhys.lloyd/Projects/sigh/node_modules/sigh-core/node_modules/baconjs/dist/Bacon.js:1047:18 at /Users/rhys.lloyd/Projects/sigh/node_modules/sigh-core/node_modules/baconjs/dist/Bacon.js:391:21 at Object.Bacon.UpdateBarrier.inTransaction (/Users/rhys.lloyd/Projects/sigh/node_modules/sigh-core/node_modules/baconjs/dist/Bacon.js:361:13) at Dispatcher.push (/Users/rhys.lloyd/Projects/sigh/node_modules/sigh-core/node_modules/baconjs/dist/Bacon.js:1227:28) at Dispatcher.handleEvent (/Users/rhys.lloyd/Projects/sigh/node_modules/sigh-core/node_modules/baconjs/dist/Bacon.js:1281:21) at /Users/rhys.lloyd/Projects/sigh/node_modules/sigh-core/node_modules/baconjs/dist/Bacon.js:6:59 at /Users/rhys.lloyd/Projects/sigh/node_modules/sigh-core/node_modules/baconjs/dist/Bacon.js:1809:19 at [object Object].sender [as _onTimeout] (/Users/rhys.lloyd/Projects/sigh/node_modules/sigh-core/node_modules/baconjs/dist/Bacon.js:2259:16) at Timer.listOnTimeout (timers.js:119:15) ! error: pipeline mocha ! Error: killed

npm ERR! Darwin 14.4.0 npm ERR! argv "/Users/rhys.lloyd/.nvm/versions/node/v0.12.7/bin/node" "/Users/rhys.lloyd/.nvm/versions/node/v0.12.7/bin/npm" "run" "test" npm ERR! node v0.12.7 npm ERR! npm v2.11.3 npm ERR! code ELIFECYCLE npm ERR! sigh@0.12.22 test: ./bootstrap.sh && sigh npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the sigh@0.12.22 test script './bootstrap.sh && sigh'. npm ERR! This is most likely a problem with the sigh package, npm ERR! not with npm itself. npm ERR! Tell the author that this fails on your system: npm ERR! ./bootstrap.sh && sigh npm ERR! You can get their info via: npm ERR! npm owner ls sigh npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request: npm ERR! /Users/rhys.lloyd/Projects/sigh/npm-debug.log

[insidewhy]

james@lonchi /tmp % node --version
v0.12.7
james@lonchi /tmp % git clone https://github.com/sighjs/sigh
Cloning into 'sigh'...
remote: Counting objects: 2362, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 2362 (delta 1), reused 0 (delta 0), pack-reused 2355
Receiving objects: 100% (2362/2362), 314.59 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1598/1598), done.
Checking connectivity... done.
james@lonchi /tmp % cd sigh 
james@lonchi sigh % npm i
------------------- SNIP --------------------
    ✓ write a single file containing a basePath
    ✓ write a single file then remove it

  27 passing (3s)
  3 pending

 * pipeline mocha complete: 5.636 seconds
 * pipeline run-tests complete: 5.637 seconds

I think it's a timing issue for that test, it has 180milliseconds of leeway which should be more than enough, but if your PC is running slow it can fail.

[insidewhy]

Gave it another 100ms in 7c8bd90 I've seen this test fail once due to that (in probably a few thousand runs but on a fast PC), this should make it more bullet proof. Please reopen if it's still not working for you.

[insidewhy]

You can check out the tests running automatically on the CI server too: https://circleci.com/gh/sighjs/sigh

[justrhysism]

I haven't got permission to reopen.

Tried the the same on my MacBook Pro Retina - 2.8 GHz Intel Core i7, 16GB RAM - initial was on an old iMac I use at work.

However, despite the significant speed boost, the tests are still failing.

I think relying on a timeout to check the results of a test is unreliable.

I'm sure everything else works fine, but as I'm looking to potentially use sigh across several platforms (Linux, OS X and Windows), generally I just run the tests to see if I'm going to have any issues.

[insidewhy]

Thanks for raising this, how many times did you try to run the test? It failed every time? This test is testing some timing related stuff. Maybe I could mock setTimeout so that the tests use fake timing!

[insidewhy]

If it fails everytime you run it, then I suspect it's something related to MacOS's file watching behaviour, I would think that chokidar might use a higher debounce on that platform or something.

I'll see if I can set up tests to run on Mac OS X via continuous integration, I don't own any Apple hardware.