format-coverage fails for simplecov 0.21.2 (minitest) with cc-test-reporter 0.10.3

[tnir]

What happens

cc-test-reporter format-coverage fails with coverage results from SimpleCov 0.21.2 and minitest 5.16.1 as follows:

$ ./cc-test-reporter format-coverage -t simplecov coverage/.resultset.json -d
DEBU[0000] coverage path coverage/.resultset.json       
DEBU[0000] using formatter simplecov                    
DEBU[0000] checking search path coverage/.resultset.json for simplecov formatter 
DEBU[0000] couldn't load branch from ENV, trying git... 
DEBU[0000] couldn't load commit sha from ENV, trying git... 
DEBU[0000] couldn't load committed at from ENV, trying git... 
DEBU[0000] Analyzing simplecov json output from latest format coverage/.resultset.json 
DEBU[0000] Analyzing simplecov json output from legacy format coverage/.resultset.json 
Error: json: cannot unmarshal object into Go struct field resultSet.coverage of type []formatters.NullInt

Why

The following results should be handled without any error:

{
  "Unit Tests": {
    "coverage": {
      "/workspace/geminabox/lib/geminabox.rb": {
        "lines": [
          null
        ]
      },
      "/workspace/geminabox/lib/geminabox/proxy.rb": {
        "lines": [
          null
        ]
      }
    },
    "timestamp": 1650000000
  },
  "Integration Tests": {
    "coverage": {
      "/workspace/geminabox/lib/geminabox.rb": {
        "lines": [
          null
        ]
      },
      "/workspace/geminabox/lib/geminabox/proxy.rb": {
        "lines": [
          null
        ]
      }
    },
    "timestamp": 1650000000
  }
}

Currently it work well with the following results:

{
  "Unit Tests": {
    "coverage": {
      "/workspace/geminabox/lib/geminabox.rb": [
          null
      ],
      "/workspace/geminabox/lib/geminabox/proxy.rb": [
          null
      ]
    },
    "timestamp": 1650000000
  },
  "Integration Tests": {
    "coverage": {
      "/workspace/geminabox/lib/geminabox.rb": [
          null
      ],
      "/workspace/geminabox/lib/geminabox/proxy.rb": [
          null
      ]
    },
    "timestamp": 1650000000
  }
}

Workaround

jq 'map_values(. | map_values(if type=="object" then map_values(.lines) else . end))' coverage/.resultset.json helped me work well.

Environments

Code Climate Test Reporter 0.10.3 (2cd3a82f34475dd7387da166ddd967fe1d713a65 @ 2022-02-15T15:28:53+0000)

• simplecov (0.21.2)
• simplecov_json_formatter (0.1.4)
• minitest (5.16.1)
• https://github.com/geminabox/geminabox/pull/475

[tnir]

Looks relevant to #488 to me.

• 488

[manicmaniac]

@tnir

Thank you for the workaround in https://github.com/codeclimate/test-reporter/issues/495#issue-1291907813

:memo: jq 'walk(try .lines // .)' coverage/.resultset.json also works for me.

[manicmaniac]

Currently simplecov formatter checks .meta.simplecov_version value in .resultset.json and changes its behavior depending whether the version is above 0.18.0.

However, apparently simplecov does not output .meta.simplecov_version to .resultset.json as far as I confirmed. If .meta.simplecov_version does not exist, the program uses legacy formatter that does not support .lines.

https://github.com/codeclimate/test-reporter/blob/22de46aa5f20cd3598fd28c76021c1a3e145bb9d/formatters/simplecov/json_formatter.go#L85 https://github.com/codeclimate/test-reporter/blob/22de46aa5f20cd3598fd28c76021c1a3e145bb9d/formatters/simplecov/json_formatter.go#L35-L46

[manicmaniac]

However, apparently simplecov does not output .meta.simplecov_version to .resultset.json as far as I confirmed.

After a further investigation, I found simplecov writes coverage/coverage.json only when CC_TEST_REPORTER_ID environment variable is set (this feature was introduced in https://github.com/simplecov-ruby/simplecov/pull/950). And cc-test-reporter format-coverage --input-type=simplecov reads coverage/coverage.json at first and fallbacks to coverage/.resultset.json if the former does not exist.

https://github.com/codeclimate/test-reporter/blob/22de46aa5f20cd3598fd28c76021c1a3e145bb9d/formatters/simplecov/simplecov.go#L12

So another possible workaround would be setting environment variable before testing.

CC_TEST_REPORTER_ID=xxx bundle exec rake test