TritonDataCenter / sdc-cloudapi

Triton Data Center Public HTTP API
Mozilla Public License 2.0
10 stars 24 forks source link

tests/runtests runner does not count failed and skipped tests #89

Open marsell opened 3 years ago

marsell commented 3 years ago

The runner says all tests passed at the end despite several failures during the test run.

Sadly, no longer have logs for this to post here.

marsell commented 3 years ago

Before (does not propagate the two failures):

[root@e5ad067e-ae5e-4505-933b-a31190be1489 (lab:cloudapi0) /opt/smartdc/cloudapi/test]# ./runtests -f users.test.js
[...]
# delete another user
ok 734 null
ok 735 should be strictly equal
ok 736 headers ok
ok 737 headers allow-origin
ok 738 headers allow-methods
ok 739 headers date
ok 740 headers request-id
ok 741 headers response time
ok 742 headers server
ok 743 headers api-version OK
# teardown
ok 744 teardown success

1..744
# tests 744
# pass  742
# fail  2

#
# TESTS COMPLETE IN 23 SECONDS, SUMMARY:
#
#   PASS: 744 / 744
#

After (propagates to summary):

[root@e5ad067e-ae5e-4505-933b-a31190be1489 (lab:cloudapi0) /opt/smartdc/cloudapi/test]# ./runtests -f users.test.js
[...]
# delete another user
ok 734 null
ok 735 should be strictly equal
ok 736 headers ok
ok 737 headers allow-origin
ok 738 headers allow-methods
ok 739 headers date
ok 740 headers request-id
ok 741 headers response time
ok 742 headers server
ok 743 headers api-version OK
# teardown
ok 744 teardown success

1..744
# tests 744
# pass  742
# fail  2

#
# TESTS COMPLETE IN 23 SECONDS, SUMMARY:
#
#   PASS: 742 / 744
#   FAIL: 2 / 744
#
# FAILED TESTS:
#
#   - test/users.test.js
#