Release 0.9.0

[noelia-lencina]

Reminder to create the tag for this release (just noticed we don't have a tag for 0.8.0.)

[filipesperandio]

@fede-moya @noelia-lencina I mean make release has an explicit call to tag

[fede-moya]

@fede-moya @noelia-lencina I mean make release has an explicit call to tag

Yeap, just saw it. You are right, weird that didn't get created last time.

[imajes]

@fede-moya without running make release, it looks like the actual binaries don't get pushed to s3, so right now the official path to latest is still 0.8.0.

Do you think you could get that run so we can go about using the update? :/ (cc @ale7714 )

[imajes]

also, on build....

formatters/simplecov/json_formatter.go:64: decoder.DisallowUnknownFields undefined (type *json.Decoder has no field or method DisallowUnknownFields)

(this may be due to a different version of go on the machine i was trying to quickly build on. You should also add GOOS and GOARCH to the make build-linux steps.... :/

[ale7714]

@imajes the binary under https://codeclimate.com/downloads/test-reporter/test-reporter-head-linux-amd64 should include this changes. Would you be open to trying this binary before we make a versioned release?

[imajes]

@ale7714 looks like it probably does, however head isn't advertised -- i assume that latest (as per the docs recommendation) is the one to use: https://docs.codeclimate.com/docs/configuring-test-coverage#section-locations-of-pre-built-binaries

[ale7714]

@imajes great question! the binaries in the documentation are the official releases. Whereas head is the latest PR merged into master which might not be ready for release.

[ale7714]

@imajes FYI, we will be pushing the binaries to S3 before end of week.

[scottohara]

I really hate to be that guy, but is there a reason that the 0.9.0 binaries weren't pushed to S3 last week?

It is almost a full year since the issue with SimpleCov 0.18+ first broke everyone's ability to upload Ruby coverage reports to CodeClimate.

It has been great to watch over the last 12 months the collaboration and effort between the SimpleCov team (particularly @PragTob) and the CodeClimate team (particularly @fede-moya) to first design, then build, and finally integrate an entirely new JSON formatter.

After all that time & effort, it just feels like we're so close.

From the outside looking in it, it seems that the only thing left to do is push the binaries, and then we can all hit Build on our respective CI/CD pipelines get back to pushing our SimpleCov reports to CodeClimate like its 2019.

A small, but welcome positive to see out what has otherwise been a horrible year for a lot of people.

A sincere thank you to everyone involved in resolving this issue; and wishing you all a better 2021 🎉 .

[ale7714]

Hey all! thank you for your patience. We have published the latest binaries for the 0.9.0 release:

• https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64
• https://codeclimate.com/downloads/test-reporter/test-reporter-0.9.0-linux-amd64

Great work @PragTob and @fede-moya for working on this 👏 !

Let us know if you have any questions!

[nextdayflight]

I think Darwin binaries have been uploaded as Linux binaries:

$ curl -OL https://codeclimate.com/downloads/test-reporter/test-reporter-0.9.0-linux-amd64
$ chmod +x test-reporter-0.9.0-linux-amd64
$ file ./test-reporter-0.9.0-linux-amd64
./test-reporter-0.9.0-linux-amd64: Mach-O 64-bit executable
$ ./test-reporter-0.9.0-linux-amd64
./test-reporter-0.9.0-linux-amd64: cannot execute binary file

[ale7714]

@nextdayflight we’re looking into it! Apologies

[ale7714]

@nextdayflight new binaries were uploaded and tested! Apologies for the issue, can you test it and confirm that everything is running for you? thank you :)

[scottohara]

Unfortunately I'm not having much luck with the new version.

I'm running this from a GitHub Actions job (which is unchanged from when it previously worked using v0.7.0 of the test reporter and SimpleCov 0.17.x).

As I understood, the new JSON formatter is automatically used by SimpleCov when the CC_TEST_REPORTER_ID environment variable is detected, which it is.

Here's the output from the GitHub Actions job:

./cc-test-reporter format-coverage --debug --input-type simplecov --output coverage/codeclimate.backend.json coverage/backend/.resultset.json
  shell: /bin/bash -e {0}
  env:
    GIT_BRANCH: master
    CC_TEST_REPORTER_ID: ***

DEBU[0000] coverage path /home/runner/work/loot/loot/coverage/backend/.resultset.json 
DEBU[0000] using formatter simplecov                    
DEBU[0000] checking search path /home/runner/work/loot/loot/coverage/backend/.resultset.json for simplecov formatter 
DEBU[0000] couldn't load committed at from ENV, trying git... 
DEBU[0000] Analyzing simplecov json output from latest format /home/runner/work/loot/loot/coverage/backend/.resultset.json  
DEBU[0000] Analyzing simplecov json output from legacy format /home/runner/work/loot/loot/coverage/backend/.resultset.json  
Error: json: cannot unmarshal object into Go struct field resultSet.coverage of type []formatters.NullInt
Usage:
  cc-test-reporter format-coverage [coverage file] [flags]

Flags:
      --add-prefix string   add this prefix to file paths
  -t, --input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, lcov-json, simplecov, xccov]
  -o, --output string       output path (default "coverage/codeclimate.json")
  -p, --prefix string       the root directory where the coverage analysis was performed (default "/home/runner/work/loot/loot")

Global Flags:
  -d, --debug   run in debug mode

Error: Process completed with exit code 255.

...and here's a short extract from the .resultset.json file that it was trying to parse: (I'm not sure if whether this is the old SimpleCov JSON format, or the new format)

{
  "RSpec": {
    "coverage": {
      "/home/runner/work/loot/loot/app/controllers/accounts_controller.rb": {
        "lines": [
          null,
          null,
          null,
          null,
          1,
          1,
          2,
          1,
          null,
          1,
          null,
          null,
          null,
          1,
          1,
          null,
          null,
          1,
          1,
          null,
          null,
          1,
          1,
          null,
          null,
          1,
          1,
          1,
          null,
          null,
          1,
          1,
          1,
          null,
          null
        ],
        "branches": {
          "[:if, 0, 7, 2, 11, 5]": {
            "[:then, 1, 8, 3, 8, 30]": 1,
            "[:else, 2, 10, 3, 10, 144]": 1
          }
        }
      }

[fede-moya]

Hey @scottohara , how are you ? Sorry for breaking your CI. I'll be digging into the problem that you are having, the information that you have already provided is enough for start debugging it I'll keep you updated.

[scottohara]

Thanks @fede-moya (no need to apologise though, I appreciate all the effort you are putting into this).

[fede-moya]

@scottohara Hey, regarding your problem.

I'll start saying that according to the .resultset.json that you shared, I understand that your GitHub action is not using SimpleCov 0.17, since at the bottom of the file there is branch coverage reported, which wasn't supported in Simplecov until 0.18. --> Check changelog https://github.com/simplecov-ruby/simplecov/blob/main/CHANGELOG.md#0180beta1-2020-01-05 . Also I can see how the output states that it's using the legacy formatter, which means that it haven't found a coverage.json file, which means that either you haven't exported your CC_TEST_REPORTER_ID variable or that your a not using latest version of simplecov. This do not matches with your first statement:

I'm running this from a GitHub Actions job (which is unchanged from when it previously worked using v0.7.0 of the test reporter and SimpleCov 0.17.x).

If your GitHub action is using either Simplecov 0.18.x or 0.19.x then is expected for the integration to not work. Could this be the case ?

[fede-moya]

@scottohara could you check that in your simplecov settings you are not overriding the default list for formatters ? If you are, please add to that list SimpleCov::Formatter::JSONFormatter.

[scottohara]

@fede-moya Aha! I know what the problem is (and it was my misunderstanding, sorry).

Firstly, I can confirm that I'm using SimpleCov 0.20.0, and that my actions build does have CC_TEST_REPORTER_ID defined.

Also I can see how the output states that it's using the legacy formatter, which means that it haven't found a coverage.json file

What I hadn't fully understood (until you mentioned this), was that when SimpleCov 0.18+ detects the CC_TEST_REPORTER_ID env var, it generates two files:

1. a legacy .resultset.json file and
2. a coverage.json file

I had (incorrectly) assumed that SimpleCov would output a single .resultset.json containing either legacy format or new JSON format.

In my action, I'm specifically telling the format-coverage command to look for a file called coverage/backend/.result.json:

./cc-test-reporter format-coverage --input-type simplecov --output coverage/codeclimate.backend.json coverage/backend/.resultset.json

(I'm passing an explicit [coverage file] argument because my coverage output is in a non-standard location)

So all I had to do was change the [coverage file] argument to coverage/backend/coverage.json, and my CI is now ✅ .

Thanks again for everything, great work!

[fede-moya]

@scottohara that's good news, I'm happy to hear that ! 👍🏼