search

codeclimate / test-reporter

Code Climate Test Reporter
MIT License
156 stars 76 forks source link

Add support for llvm.coverage.json.export format #439

Closed paulofaria closed 3 years ago

paulofaria commented 4 years ago

This PR adds support for the llvm.coverage.json.export format. This is the format generated by the Swift Package Manager. The commands below allow you to generate the coverage file:

mkdir TestCoverage
cd TestCoverage
swift package init
swift test --enable-code-coverage  
swift test --show-codecov-path

The last command shows the path of the generated coverage file. I added a new formatter called lcov-json which can be used like so:

test-reporter format-coverage $(swift test --show-codecov-path) -t lcov-json
CLAassistant commented 4 years ago

CLA assistant check
All committers have signed the CLA.

paulofaria commented 4 years ago

@filipesperandio Hey, Filipe. Hope everything is well with you and your loved ones. Can you please take a look at the code and let me know if there is anything I need to rework? Thank you!

f-moya commented 4 years ago

@filipesperandio It works as described by @paulofaria, attaching screenshot. Got it working with

cc-test-reporter before-build
cc-test-reporter format-coverage coverage/swift-llvm-cov-test.json -t lcov-json
cc-test-reporter upload-coverage

Couldn't get it to work with just cc-test-reporter before-build and cc-test-reporter after-build

Screen Shot 2020-09-07 at 19 16 40
f-moya commented 4 years ago

could you @paulofaria got it to working using the after-build command ?

filipesperandio commented 4 years ago

@fede-moya Is there an error with the after-build?

f-moya commented 4 years ago

yeap, there is an error.

swift-llvm-cov-test >> test-reporter after-build -t lcov-json
ERRO[0000] could not find coverage file
could not find any files in search paths for lcov-json. search paths were:
Error: could not find any files in search paths for lcov-json. search paths were:
Usage:
  cc-test-reporter after-build [flags]

Flags:
  -s, --batch-size int               batch size for source files (default 500)
  -e, --coverage-endpoint string     endpoint to upload coverage information to (default "https://api.codeclimate.com/v1/test_reports")
  -t, --coverage-input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, lcov-json, simplecov, xccov]
      --exit-code int                exit code of the test run
  -r, --id string                    reporter identifier (default "41241242141")
      --insecure                     send coverage insecurely (without HTTPS)
  -p, --prefix string                the root directory where the coverage analysis was performed (default "/Users/federicomoya/Documents/repos/swift-llvm-cov-test")

Global Flags:
  -d, --debug   run in debug mode
swift-llvm-cov-test >> ls
Package.swift README.md     Sources       Tests         coverage
swift-llvm-cov-test >> ls coverage
swift-llvm-cov-test.json
filipesperandio commented 4 years ago

Thanks @fede-moya . @paulofaria Any idea?

paulofaria commented 4 years ago

There is no default search paths for coverage files generated by the Swift Package Manager. As you can see I removed the default search path in this commit (the one I removed was a leftover from the formatter I used as a base for this formatter). The Swift Package Manager code coverage feature was designed to be used with swift test --show-codecov-path, so I don't think it's possible to use it with the after-build command.

paulofaria commented 4 years ago

Just inspected the directory of the code coverage file created by SwiftPM. Running the following command:

swift test --show-codecov-path

Gives us this output in my machine:

/Users/paulo/Development/Playgrounds/TestCoverage/.build/x86_64-apple-macosx/debug/codecov/TestCoverage.json

Inspecting the directory of the file with: 

ls /Users/paulo/Development/Playgrounds/TestCoverage/.build/x86_64-apple-macosx/debug/codecov

We get:

TestCoverage.json                    default1567809077063180439_0.profraw
default.profdata

Does default search path support globbing? If so, we could try to match any .json file inside a codecov directory.

paulofaria commented 4 years ago

I should warn that, since this approach assumes implementation details of SwiftPM, it is not future-proof. SwiftPM might change the directory and filename of the code coverage file.

paulofaria commented 4 years ago

Looks like the CI error is unrelated to the changes in this PR.

/usr/local/Homebrew/Library/Homebrew/brew.rb:16:in `<main>': HOMEBREW_REQUIRED_RUBY_VERSION was not exported! Please call bin/brew directly! (RuntimeError)
f-moya commented 3 years ago

@filipesperandio @paulofaria this commit fixes the CI. It should get merge soon.

Considering that according to @paulofaria due to the specifics of swift the after build command doesn't make much sense, having just the option to use the reporter with the before build command sounds good to me. Although we should document that.

I think after the fix of the CI gets in, we could merge this one. WDYT @filipesperandio ?

noelia-lencina commented 3 years ago

@fede-moya @filipesperandio pinging you as you have been following this PR closer. The CI update did indeed fix the failing CI in this PR, do you think this is safe to merge or more steps are needed?

f-moya commented 3 years ago

@fede-moya @filipesperandio pinging you as you have been following this PR closer. The CI update did indeed fix the failing CI in this PR, do you think this is safe to merge or more steps are needed?

Nice, I'm good with how it is now. The only thing would be for us to add a ticket to mention this new option in the docs, and it's particular use, like mentioning that there is no expected path.