search

kiwi-bdd / Kiwi

Simple BDD for iOS
BSD 3-Clause "New" or "Revised" License
4.14k stars 512 forks source link

Tests are executed in a wrong order under certain conditions (Xcode 8.3, single suite) #692

Open stigger opened 7 years ago

stigger commented 7 years ago

Xcode 8.3 sometimes orders the tests within a suite alphabetically. If two different tests have clashing prefixes, this leads to the incorrect execution order.

SPEC_BEGIN(SpecWithSameNames)
describe(@"desc", ^{
    specify(^{
        NSLog(@"1");
        [[@"" should] beEmpty];
    });
    it(@"it", ^{
        NSLog(@"2");
        [[@"" should] beEmpty];
    });
}),
describe(@"desc", ^{
    specify(^{
        NSLog(@"3");
        [[@"" should] beEmpty];
    });
});
SPEC_END

Run as usual, everything is correct:

Test Suite 'Selected tests' started at 2017-04-04 17:11:30.008
Test Suite 'Tests-xctest.xctest' started at 2017-04-04 17:11:30.009
Test Suite 'SpecWithSameNames' started at 2017-04-04 17:11:30.009
Test Case '-[SpecWithSameNames Desc_]' started.
2017-04-04 17:11:30.111682+0200 xctest[42445:6427702] DTKPLogging: Enabled at level: 3
2017-04-04 17:11:30.111906+0200 xctest[42445:6427702] DTTapLogging: Enabled at level: 3
2017-04-04 17:11:30.193003+0200 xctest[42445:6427677] 1
2017-04-04 17:11:30.193208+0200 xctest[42445:6427677] + 'desc, should be empty' [PASSED]
Test Case '-[SpecWithSameNames Desc_]' passed (0.184 seconds).
Test Case '-[SpecWithSameNames Desc_It]' started.
2017-04-04 17:11:30.195031+0200 xctest[42445:6427677] 2
2017-04-04 17:11:30.195136+0200 xctest[42445:6427677] + 'desc, it' [PASSED]
Test Case '-[SpecWithSameNames Desc_It]' passed (0.001 seconds).
Test Case '-[SpecWithSameNames Desc_2]' started.
2017-04-04 17:11:30.196289+0200 xctest[42445:6427677] 3
2017-04-04 17:11:30.196350+0200 xctest[42445:6427677] + 'desc, should be empty' [PASSED]
Test Case '-[SpecWithSameNames Desc_2]' passed (0.001 seconds).
Test Suite 'SpecWithSameNames' passed at 2017-04-04 17:11:30.197.
     Executed 3 tests, with 0 failures (0 unexpected) in 0.186 (0.187) seconds
Test Suite 'Tests-xctest.xctest' passed at 2017-04-04 17:11:30.197.
     Executed 3 tests, with 0 failures (0 unexpected) in 0.186 (0.188) seconds

Now go to the test navigator end click the "play" button next to "SpecWithSameNames":

Test Suite 'Selected tests' started at 2017-04-04 17:13:21.374
Test Suite 'Tests-xctest.xctest' started at 2017-04-04 17:13:21.376
Test Suite 'SpecWithSameNames' started at 2017-04-04 17:13:21.376
Test Case '-[SpecWithSameNames Desc_]' started.
2017-04-04 17:13:21.479547+0200 xctest[42484:6429066] DTKPLogging: Enabled at level: 3
2017-04-04 17:13:21.479604+0200 xctest[42484:6429066] DTTapLogging: Enabled at level: 3
2017-04-04 17:13:21.553235+0200 xctest[42484:6429018] 1
2017-04-04 17:13:21.553425+0200 xctest[42484:6429018] + 'desc, should be empty' [PASSED]
Test Case '-[SpecWithSameNames Desc_]' passed (0.177 seconds).
Test Case '-[SpecWithSameNames Desc_2]' started.
2017-04-04 17:13:21.555759+0200 xctest[42484:6429018] 3
2017-04-04 17:13:21.555890+0200 xctest[42484:6429018] + 'desc, should be empty' [PASSED]
Test Case '-[SpecWithSameNames Desc_2]' passed (0.001 seconds).
Test Case '-[SpecWithSameNames Desc_It]' started.
2017-04-04 17:13:21.556725+0200 xctest[42484:6429018] 2
2017-04-04 17:13:21.556782+0200 xctest[42484:6429018] + 'desc, it' [PASSED]
Test Case '-[SpecWithSameNames Desc_It]' passed (0.001 seconds).
Test Suite 'SpecWithSameNames' passed at 2017-04-04 17:13:21.557.
     Executed 3 tests, with 0 failures (0 unexpected) in 0.179 (0.181) seconds
Test Suite 'Tests-xctest.xctest' passed at 2017-04-04 17:13:21.557.
     Executed 3 tests, with 0 failures (0 unexpected) in 0.179 (0.181) seconds