Generate coverage data from each of the tests

[ollieatkinson]

Issue type: Feature Request

Why:

I need to gather coverage data from running tests.

How:

This is available in xcodebuild using the following:

xcodebuild \
    -derivedDataPath /tmp/some/directory \
    -enableCodeCoverage YES \
   -destination 'platform=iOS Simulator,name=iPhone 6,OS=10.0' \
   -xctestrun path/to/file.xctestrun \
   test-without-building

to extract the coverage data with LLVM

xcrun llvm-cov report -instr-profile /tmp/some/directory/Build/Intermediates/CodeCoverage/Coverage.profdata /tmp/some/directory/Build/Intermediates/CodeCoverage/Products/Debug-iphonesimulator/Example.app/Example

I don't know if its possible to pass xcodebuild those command line arguments using pxctest or if it is something that fbxctest would need to incorporate (or neither).

[plu]

We would need to check if FBSimulatorControl supports enabling code coverage. Otherwise reverse engineer what xcodebuild does when passing -enableCodeCoverage yes, and try to replicate that.

[ollieatkinson]

specifying these configuration arguments does that:

GCC_GENERATE_TEST_COVERAGE_FILES = YES
GCC_INSTRUMENT_PROGRAM_FLOW_ARCS = YES

[plu]

These sound like compiler flags, I am still wondering how that fits into test-without-building, which is just executing the tests.

[ollieatkinson]

You are right, my bad. That is just for compiling. The documentation suggests that its a scheme setting,

     -enableCodeCoverage [YES | NO]
           Turns code coverage on or off during testing. This overrides the setting for the test action of a scheme in a workspace.

I wonder if there is anything in the xctestrun file we can enable, looking at its contents I can see its already specifying the following:

        <key>ClangProfileDataGenerationFilePath</key>
        <string>__DERIVEDDATA__/Build/Intermediates/CodeCoverage/ProfileData</string>
        <key>ClangProfileFilePath</key>
        <string>__DERIVEDDATA__/Build/Intermediates/CodeCoverage/Coverage.profdata</string>

[plu]

Looks like we'd need to read these two properties:

https://github.com/plu/FBSimulatorControl/blob/pxctest/PrivateHeaders/IDEFoundation/IDETestRunSpecification.h#L73-L74

And then pass them down the road in FBSimulatorControl, somewhere around here it would start:

https://github.com/plu/FBSimulatorControl/blob/pxctest/XCTestBootstrap/Configuration/FBXCTestRun.m#L156

It's a bit of work, I'm not sure I will find time for this anytime soon. Are you interested in looking into it?

Right now I still have a pending pull request open on FBSimulatorControl, which the work needs to be based off: https://github.com/facebook/FBSimulatorControl/pull/340

[ollieatkinson]

I can take a look - but specifying just those two arguments isn't enough. -enableCodeCoverage YES must be doing some more magic.

[ollieatkinson]

Still haven't had luck - I noticed xctool have investigated this also. I will keep it on my back burner.

ref: https://github.com/facebook/xctool/issues/574

[plu]

It's just a matter of finding them in derivedData, no? In our discussion in January we figured out that -enableCodeCoverage is a flag that needs to be passed to xcodebuild when we do build-for-testing. This should then already generate coverage data, but it will be located in Xcode's derivedData path. So the quick win would be to look there (without changing anything pxctest).

[ollieatkinson]

Not quite, building the tests with -enableCodeCoverage YES does indeed add some additional parameters into the .xctestrun file - however this does not automatically generate the coverage when run.

This fails:

$ xcodebuild \
    -derivedDataPath "$PWD/derivedData" \
    -enableCodeCoverage YES \
    -scheme 'MyApp' \
    -workspace 'MyApp.xcworkspace' \
    -destination 'platform=iOS Simulator,name=iPhone 7,OS=10.2' \
    build-for-testing
$ xcodebuild \
    -derivedDataPath "$PWD/derivedData" \
    -destination 'platform=iOS Simulator,name=iPhone 7,OS=10.2' \
    -xctestrun derivedData/Build/Intermediates/CodeCoverage/Products/MyApp_iphonesimulator10.2-x86_64.xctestrun \
    test-without-building
$ xcrun llvm-cov report \
    -instr-profile derivedData/Build/Intermediates/CodeCoverage/Coverage.profdata 
    derivedData/Build/Intermediates/CodeCoverage/Products/Debug-iphonesimulator/MyApp.app/MyApp

Where appending -enableCodeCoverage YES to the test-without-building command works - so I think its an issue elsewhere.

$ xcodebuild \
    -derivedDataPath "$PWD/derivedData" \
    -enableCodeCoverage YES \
    -scheme 'MyApp' \
    -workspace 'MyApp.xcworkspace' \
    -destination 'platform=iOS Simulator,name=iPhone 7,OS=10.2' \
    build-for-testing
$ xcodebuild \
    -derivedDataPath "$PWD/derivedData" \
    -enableCodeCoverage YES \
    -destination 'platform=iOS Simulator,name=iPhone 7,OS=10.2' \
    -xctestrun derivedData/Build/Intermediates/CodeCoverage/Products/MyApp_iphonesimulator10.2-x86_64.xctestrun \
    test-without-building
$ xcrun llvm-cov report \
    -instr-profile derivedData/Build/Intermediates/CodeCoverage/Coverage.profdata \
    derivedData/Build/Intermediates/CodeCoverage/Products/Debug-iphonesimulator/MyApp.app/MyApp

So I agree that the best thing to do would not be to change anything in pxctest and figure this out.

[plu]

Thank you for providing the xcodebuild examples, now it's clear to me what is missing 👍 .

[plu]

I just pushed a change (8c16f86f8d23fe2cf7979d6fc83b5828a673be70) that will generate the (raw) coverage data. You still need to process them, documentation can be found here: https://clang.llvm.org/docs/SourceBasedCodeCoverage.html

xcrun llvm-profdata merge -sparse foo1.profraw foo2.profraw -o foo3.profdata

Once that is done, you can generate the report like you did via xcrun llvm-cov report ...

[plu]

The raw coverage data you can find inside the different targets/devices in the output directory.

[plu]

One more thing: If you don't want to update to master of pxctest, you can try this instead with the current version:

$ export PXCTEST_CHILD_LLVM_PROFILE_FILE="/tmp/test-coverage.%p.profraw"
$ export PXCTEST_CHILD___XPC_LLVM_PROFILE_FILE="/tmp/test-coverage.%p.profraw"
$ pxctest ...

[ollieatkinson]

Genius! I will close this now.