SlatherOrg / slather

Generate test coverage reports for Xcode projects & hook it into CI.
MIT License
1.56k stars 238 forks source link

Slather code coverage doesn't work on -release build : Malformed coverage data #438

Closed gauravdalal closed 4 years ago

gauravdalal commented 5 years ago

Hello,

I have been trying to integrate Slather with our app on Jenkins. It works perfectly when I run it for Debug version but it certainly doesn't work for Release version. The job fails as it looks for binary file in "ProfileData" directory when running for Release version. I am running it via command line (not .yml). Also I'm copying the contents of DerivedData to a custom location and using --build-directory from there only.

Command I'm using is: slather coverage --html --output-directory "Directory-in-jenkins-workspace" --build-directory /Users/developer/Artifact/development/ --binary-basename Org.app --scheme 'Org' --workspace /Users/developer/Documents/TEST/code/Org.xcworkspace /Users/developer/Documents/TEST/code/Org.xcodeproj/

09:29:01 Slathering... 09:29:02 /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/project.rb:559:infind_binary_files': No product binary found in /Users/developer/Artifact/development/Build/ProfileData/FA94A549-4432-479B-B651-989ABF4F7962. (StandardError) 09:29:02 from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/project.rb:433:in configure_binary_file' 09:29:02 from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/project.rb:315:inconfigure' 09:29:02 from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/command/coverage_command.rb:57:in execute' 09:29:02 from /usr/local/lib/ruby/gems/2.5.0/gems/clamp-1.3.1/lib/clamp/command.rb:66:inrun' 09:29:02 from /usr/local/lib/ruby/gems/2.5.0/gems/clamp-1.3.1/lib/clamp/subcommand/execution.rb:18:in execute' 09:29:02 from /usr/local/lib/ruby/gems/2.5.0/gems/clamp-1.3.1/lib/clamp/command.rb:66:inrun' 09:29:02 from /usr/local/lib/ruby/gems/2.5.0/gems/clamp-1.3.1/lib/clamp/command.rb:140:in run' 09:29:02 from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/bin/slather:17:in<top (required)>' 09:29:02 from /usr/local/bin/slather:23:in load' 09:29:02 from /usr/local/bin/slather:23:in

' 09:29:02 No product binary found in /Users/developer/Artifact/development/Build/ProfileData/FA94A549-4432-479B-B651-989ABF4F7962. 09:29:02 09:29:02 Are you sure your project is generating coverage? Make sure you enable code coverage in the Test section of your Xcode scheme. 09:29:02 Did you specify your Xcode scheme? (--scheme or 'scheme' in .slather.yml) 09:29:02 If you're using a workspace, did you specify it? (--workspace or 'workspace' in .slather.yml) 09:29:02 If you use a different Xcode configuration, did you specify it? (--configuration or 'configuration' in .slather.yml) 09:29:02 09:29:02 Build step 'Execute shell' marked build as failure`

gauravdalal commented 5 years ago

figured that I wasn't passing the --configuration argument in the slather command. But after adding this and with few other changes it started to check binary from correct location, i.e Release-iphonesimulator. But the problem is changed to "Malformed coverage data".

Little bit of background for context:

Here is the command I'm using and following is the slather results.

XcodeBuild Command: xcodebuild clean build-for-testing -enableCodeCoverage YES -workspace 'Org.xcworkspace' -scheme 'Org' -sdk iphonesimulator12.2 -ONLY_ACTIVE_ARCH=YES ENABLE_TESTABILITY=YES -configuration Release VALID_ARCHS='x86_64' SWIFT_OPTIMIZATION_LEVEL='-Onone'

Test Run Command: xcodebuild test-without-building -enableCodeCoverage YES "$testClass" -xctestrun "$artifactName" -derivedDataPath "/Users/developer/Artifact/$AppVersion/" -destination 'platform=iOS Simulator,name=iPhone 8,OS=latest' -resultBundlePath $testReportDir | tee $WORKSPACE/testlog.log | xcpretty --report html , --simple , --output $WORKSPACE/testreport.html

Slather Command: slather coverage --verbose --html --output-directory CoverageReport/ --build-directory /Users/developer/Artifact/development/ --binary-basename 'Org.app' --configuration 'Release' --scheme 'Org' /Users/developer/Documents/TEST/code/Org.xcodeproj/

13:00:19 Slathering...
13:00:21 Processing coverage file: /Users/developer/Artifact/development/Build/ProfileData/D25406D6-6DF7-42E2-B86B-599D81E9CFE1/Coverage.profdata
13:00:21 Against binary files:
13:00:21     /Users/developer/Artifact/development/Build/Products/Release-iphonesimulator/Org.app/Org 
13:00:22 error: /Users/developer/Artifact/development/Build/Products/Release-iphonesimulator/Org.app/Org: Failed to load coverage: Malformed instrumentation profile data
13:00:22 error: Could not load coverage information
13:00:22 /usr/local/lib/ruby/gems/2.5.0/gems/json-2.2.0/lib/json/common.rb:156:in `parse': 785: unexpected token at '' (JSON::ParserError)
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/json-2.2.0/lib/json/common.rb:156:in `parse'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/project.rb:135:in `pathnames_per_binary'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/project.rb:124:in `block in profdata_coverage_files'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/project.rb:123:in `each'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/project.rb:123:in `profdata_coverage_files'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/project.rb:98:in `coverage_files'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/coverage_service/html_output.rb:26:in `post'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/command/coverage_command.rb:97:in `post'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/lib/slather/command/coverage_command.rb:59:in `execute'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/clamp-1.3.1/lib/clamp/command.rb:66:in `run'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/clamp-1.3.1/lib/clamp/subcommand/execution.rb:18:in `execute'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/clamp-1.3.1/lib/clamp/command.rb:66:in `run'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/clamp-1.3.1/lib/clamp/command.rb:140:in `run'
13:00:22    from /usr/local/lib/ruby/gems/2.5.0/gems/slather-2.4.7/bin/slather:17:in `<top (required)>'
13:00:22    from /usr/local/bin/slather:23:in `load'
13:00:22    from /usr/local/bin/slather:23:in `<main>'
13:00:22 Build step 'Execute shell' marked build as failure`
gauravdalal commented 5 years ago

@ksuther - can you suggest some help on this Malformed instrumentation profile data? It would be great help. Thanks!

ksuther commented 4 years ago

That error comes from llvm-cov directly. You might try manually running llvm-cov on the coverage data that is created and see if that works or gives you any clues.

gauravdalal commented 4 years ago

Thanks! It had some issue with static frameworks we were using in our project where the llvm-cov was failing. We resolved this issue by creating another configuration with similar settings as Debug for those frameworks.