search

SlatherOrg / slather

Generate test coverage reports for Xcode projects & hook it into CI.
MIT License
1.56k stars 238 forks source link

Incorrect code coverage: Not running against all files #340

Open iwasrobbed-ks opened 7 years ago

iwasrobbed-ks commented 7 years ago

Xcode version: 9 GM

It appears Slather is only calculating coverage for the files it actually has tests for. This gives an unrealistically high code coverage % since it completely ignores all the untested code within the project.

Additionally, even the tested files have incorrect coverage #'s. All the 100% files shown by Slather below are wrong.

Xcode gives 85.8% whereas Slather gives 99.5%

screen shot 2017-09-22 at 1 28 30 pm 
SwitchboardTests/Helpers/TestAnalyticsProvider.swift: 12 of 15 lines (80.00%)
SwitchboardTests/SwitchboardCacheTests.swift: 26 of 26 lines (100.00%)
SwitchboardTests/SwitchboardDefaultStoreTests.swift: 11 of 11 lines (100.00%)
SwitchboardTests/SwitchboardExperimentTests.swift: 229 of 229 lines (100.00%)
SwitchboardTests/SwitchboardFeatureTests.swift: 70 of 70 lines (100.00%)
SwitchboardTests/SwitchboardPropertiesTests.swift: 13 of 13 lines (100.00%)
SwitchboardTests/SwitchboardSubclassTests.swift: 14 of 14 lines (100.00%)
SwitchboardTests/SwitchboardTests.swift: 213 of 213 lines (100.00%)
Test Coverage: 99.49%
jeffcampbell commented 6 years ago

I am seeing this issue as well.

krin-san commented 6 years ago

Having the same issue on my projects. Slather shows 28.78% while only 16.24% with xcode.

YufeiG commented 6 years ago

I'm also seeing a difference between Xcode % and Slather %. I am wondering if this is due to Xcode (at least on 9.3) counting partial line coverage correctly, while Slather considers a line covered if any part of the line is covered?

hbursk commented 6 years ago

I'm seeing only a few files included in my coverage rather than all the files that don't have coverage. Here's my .slather.yml. This results in coverage of about 65% rather than something below 15%.

coverage_service: cobertura_xml xcodeproj: MyProject.xcodeproj workspace: MProject.xcworkspace source_directory: source/* output_directory: xml_report scheme: MyProject ignore:

Update: Tested using CircleCI Xcode 9.2, and I'm seeing the same issue. Slather code coverage results at 54% while coverage in Xcode shows closer to 13%.

Update: My previous project with an issue used Objective-C and was generating gcno coverage data. Testing with a Swift project seemed to work fine. Would still like this to work with my Objective-C code.