Failing tests are sometimes missing from JUnit report uploaded to test observability server

[lawrence-forooghian]

For example, see https://github.com/ably/ably-asset-tracking-swift/actions/runs/3679080639/jobs/6235577712; the following tests are failing (and many others):

• ChannelModesTests.testShouldCreateOnlyOnePublisherAndOneSubscriberConnection()
• PublisherAndSubscriberSystemTests.testSubscriberReceivesAssetConnectionStatusWithBicycleProfile()
• PublisherAndSubscriberSystemTests.testSubscriberReceivesAssetConnectionStatusWithCarProfile()
• PublisherAndSubscriberSystemTests.testSubscriberReceivesPublisherMessageWithBicycleProfile()
• PublisherAndSubscriberSystemTests.testSubscriberReceivesPublisherMessageWithCarProfile()

But none of those appear in the JUnit report uploaded to the test observability server.

Also, the number of tests per upload seems to be varying wildly (see the "Total number of tests" column here).

[sync-by-unito[bot]]

➤ Automation for Jira commented:

The link to the corresponding Jira issue is https://ably.atlassian.net/browse/SDK-3180

[lawrence-forooghian]

I've pushed the code with the failures to https://github.com/ably/ably-asset-tracking-swift/tree/513-example-of-missing-tests-in-junit-report so we can reproduce this later. It's happening because the tests are causing a nil implicitly unwrapped optional to be unwrapped, and the test process is crashing. This seems to be an issue in how xcpretty is converting the xcodebuild output that's piped in to it. I'll upload an example of this xcodebuild output later.

[lawrence-forooghian]

OK, here's an example of an xcodebuild log, and the corresponding .xcresult bundle. Both from my machine.

• xcodebuild.log
• Test-ably-asset-tracking-swift-Package-2022.12.13_19-26-26--0300.xcresult