search

wshowair / ikea-shopping-list

http://waelshowair.com
0 stars 0 forks source link

Code Coverage is not deterministic #1

Open wshowair opened 8 years ago

wshowair commented 8 years ago

Platform : Simulator/Real Device Version : 1.0.0

Steps to reproduce

  1. Checkout code version 8353dce from master branch
  2. Make sure that Code Coverage is enabled as explained here.

Expected Results

Code coverage accumulates all results of code testing from different test targets

Actual Result

Message from debugger: Terminated due to signal 9

wshowair commented 8 years ago

After spending 10 hours trying to get code coverage for UI Test cases, I found it is something non-deterministic at all. Most of the time the coverage percent does not increase, just few time it did.

I tried to run UI Tests only without unit tests and found that, the coverage is zero !! as indicated in the image below.

551564433-screen shot 2015-11-07 at 09 06 00

Also I found out this recent post on apple developer forum. I reported a bug for Apple No. 23453311

wshowair commented 8 years ago

I made a simple trial,

  1. create a project with NavBarContrl, FirstTabelViewController & SecondTableViewController, keep the unit tests as default
  2. Leave unit tests, add one record in UI Tests to navigate from first table view controller to the second.
  3. Make sure code coverage is enabled
  4. Run Tests then check code coverage, it is 40% !!

I opened a dedicated stackoverflow question

wshowair commented 8 years ago

I tried to run the tests on Real Phone to exclude the possibility that this is a simulator bug but it did not work. I just noticed that the probability of high coverage percent was higher

wshowair commented 8 years ago

I would like to document some of the different errors that i encounter while failing to generate accurate code coverage report:

  1. SIGNAL 9 is received 551564433-screen shot 2015-11-07 at 09 06 00
  2. Could not load any Objective-C class information 551564433-screen shot 2015-11-07 at 09 06 00