Support additional coverage output formats

[devstuff]

Expected Behavior

Support additional output formats for the coverage report.

• IDE-specific integration format (e.g. for IntelliJ). Useful for exposing coverage when using the IDE plugin
• CI tooling-specific format (e.g. Sonarqube's generic XML coverage format).

Actual Behavior

Only JSON output is available.

Steps to Reproduce the Problem

Additional Info

[anderseknert]

For the CI/CD case it's probably OK to use an additional tool for format conversion, like how this script converts OPA test results from JSON into JUnit XML.

I don't know what IntelliJ IDEA expects here.. @vgramer , is this something you've looked into?

[vgramer]

@anderseknert I take a quick look at how implements coverage in IDEA. It seems that it takes any file in input and it's the responsibility of the plugin to parse it into an object that IDEA can handle. So I don't think any updates are required on OPA

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.

[sidpremkumar]

I would like to bump / ask if there is any updates on this thread? We would love to push coverage data to codebuild, but not sure if opa generates anything standard that could be pushed.

As a reference here is a list of supported formats (under reports): https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#reports-buildspec-file

[anderseknert]

I think the problem with "anything standard" is that there are just too many to choose from 😄 Would something like the JUnit output approach work here as well, i.e. a script outside of OPA that did the transformation from JSON to whatever output format desired?

[sidpremkumar]

I think the problem with "anything standard" is that there are just too many to choose from 😄 Would something like the JUnit output approach work here as well, i.e. a script outside of OPA that did the transformation from JSON to whatever output format desired?

Yes! that would also be perfect, I just could not find anything that would work with what AWS is expecting, but if there is something I missed please let me know!

[anderseknert]

SimpleCov was the only JSON format in that list of supported AWS CodeBuild formats, so I got curious what such a translation might look like. I've submitted the result of my experiment to the contrib repo.. feel free to try it out! And let me know if it works 😅

[vgramer]

TLDR: I think opa coverage json output is enough and work should be done on the IDE plugin.

IIRC, IDEA has its own internal format for representing code coverage. The coverage JSON output of OPA contains all the information needed and just needs to be "translated" to IDEA internal format ( matching rego files to virtual document, adapt line numbers...).

The most complicated part is creating a specific runner for the run configuration( it may require some refactoring of the current code of the running configuration). I also have to check if the coverage API is part of IDEA community version or ultimate.

[anderseknert]

Would love to see it integrated in the IDEA plugin @vgramer! 😃

[sidpremkumar]

@anderseknert you are the best!!! Can confirm now I can export to Codebuild coverage reports following the instructions on the PR you mentioned:

Thank you so much :)

[anderseknert]

Awesome! Thanks for letting me know, @sidpremkumar 👍

[anderseknert]

As shown above, and given how IDEA likely can work with that we have, I think the way forward here is not to add additional formats to OPA core, but to allow transforms of the current output format as utility scripts and have them collected in the contrib repo (or linked from external repositories). @devstuff if you had some particular output format in mind, like the SonarQube one, I'd be happy to work with you on a conversion script.

Examples of what can be done exists already for:

• JUnit
• SimpleCov

I'd love to see more added to the list!

One thing that we possibly might want to improve on though is the current output format, if it turns out to limit what we're able to transform it into.

Closing for now.

[sidpremkumar]

Hate to bring this thread back, but it looks like there are issues pushing the data to AWS, I've contacted AWS support as I thought it might be an issue on their end, but they got back to me with this:

Please allow me to explain you that based on the git link [1] shared internal team noticed you are using a report generated by SimpleCov-JSON. However, as you might already be aware CodeBuild accepts JSON code coverage reports generated by simplecov, not simplecov-json [2] .Hence, they suggested to use simplecov [3] instead to generate reports in code build. Further, they have also tested using a CodeCov example provided here [4] and the builds are successful and the report was generated successfully. 

To produce the coverage report, ran a 'bundle exec rake'. This created a "coverage/.resultset.json" file which was then referenced in the buildspec.yml [5]:

——————
reports:
CodeCoverage:
files:
- 'coverage/.resultset.json'
file-format: SimpleCov
———————

Therefore, I would suggest you to please test this using a report generated by simplecov [3] and verify it is results.

I can try to help in anyway I can ,but @anderseknert would you happen to know if this is an issue you are aware of?

The coverage is always being reported as 100% in AWS (even when manually removing tests :( )

[anderseknert]

Hi @sidpremkumar ! That's interesting - the JSON provided seems to be exactly what the project recommends for JSON output for SimpleCov.

If you could provide me with an example of a file that they'd accept, I'd be happy to help create an output transformer for that.

[sidpremkumar]

Following the instructions that they sent:

• Clone this repo: https://github.com/codecov/example-ruby
• Run bundle exec rake The coverage file it generated was:

  {
  "Unit Tests": {
  "coverage": {
    "/Users/sidpremkumar/Documents/BastionZero/Random/example-ruby/lib/awesome.rb": {
      "lines": [
        null,
        null,
        1,
        1,
        1,
        null,
        null,
        1,
        1,
        null,
        null,
        1,
        0,
        null,
        null
      ]
    }
  },
  "timestamp": 1652099838
  }
  }

But to be honest, I've spent a bit comparing that with yours and I can't seem to find much of a difference. This is the output of the script that you created:

{
  "coverage": {
    "somefile.rego": {
      "lines": [
        null,
        null,
        null,
        null,
        1,
        null,
        null,
        1,
        1,
        1,
        null,
        null,
        1,
        1,
        1,
        1,
        null,
        null,
        1,
        1,
        1,
        1,
        null,
        null,
        1,
        1,
        1,
        1
      ]
    }
}

Its possible it could be the hierarchy in the dict? i.e. a overall key called 'Unit Tests' and then a key called 'coverage'?

[anderseknert]

If that's the only difference, you should be able to modify the transformer policy to something like:

from_opa := {
    "Unit Tests": {
        "coverage": coverage,
        "timestamp": round(time.now_ns() / 1000000000)
    }
}

Let me know if they'll accept that :)