incorrect (zero) error status: when multiple urls are tested

[romanchyla]

when there are multiple url in mocha_phantomjs configuration, the run is always reported as successful; despite one (or multiple) url failing

is there an option to 'fail fast'? (or at least report non zero status when one of the urls is non-zero?)

here is an example, the test:full_testing will always succeed, but individual tests will fail

mocha_phantomjs: {
      options: {
        //'reporter': 'progress',
        'output': 'test/reports/' + (grunt.option('testname') || 'mocha/discovery')
      },

      local_testing: ['test/' + (grunt.option('testname') || 'mocha/discovery') + '.spec.html'],

      web_testing: {
        options: {
          urls: [
              'http://localhost:<%= local.port || 8000 %>/test/' + (grunt.option('testname') || 'mocha/discovery') + '.spec.html'
          ]
        }
      },

      full_testing: {
        options: {
          urls: [
            'http://localhost:<%= local.port || 8000 %>/test/mocha/discovery.spec.html',
            'http://localhost:<%= local.port || 8000 %>/test/mocha/discovery-ui.spec.html',
            'http://localhost:<%= local.port || 8000 %>/test/mocha/discovery-qb.spec.html'
          ]
        }
      }
    },

you can see it at work https://travis-ci.org/adsabs/bumblebee/jobs/33810821 - line 2400

[romanchyla]

the log is for 0.4.0, but I've tested also against the latest 0.6.0 release - both finish with 'Done, without errors'

[jdcataldo]

This is actually correct if you are using the output option. https://github.com/jdcataldo/grunt-mocha-phantomjs#output

If your ci isn't verifying against the outputted file then you should remove output and the exit code will fire if a test fail letting travis know the tests failed.

[romanchyla]

thanks a lot!