search

thaliproject / CI

CI project for testing mobile devices
MIT License
2 stars 3 forks source link

Tests claim to pass when they haven't #80

Closed yaronyg closed 8 years ago

yaronyg commented 8 years ago

See thaliproject/Thali_CordovaPlugin#956. The test output said:

Test 83268893919c9ad(919c9ad) has successfully finished without an error

See https://github.com/ThaliTester/TestResults/tree/83268893919c9ad_added_multiConnect__multiConnectResolved__multiConnectConnectionFailure_baydet/ for the logs

But if you click on the link and look at the iOS summary results you will see that they say false.

How can iOS be failing but the test says it passed? What's going on?

larryonoff commented 8 years ago

I checked iOS code and it looks that iOS reports properly.

yaronyg commented 8 years ago

Exactly but CI still reports a success even though iOS reported a failure! This is super bad.

czyzm commented 8 years ago

For CI a failure means a _TEST_LOGGER:[PROCESS_ON_EXIT_FAILED]_\ in logs. The issue is that the tests are failing but they report _TEST_LOGGER:[PROCESS_ON_EXIT_SUCCESS]_.

This is because there is a temporary code in UnitTest_app.js that is logging success and returning in order to omit the node tests. The bad result of native tests would be processed later.

For now, until we enable node tests we need to log the failure immediately.