search

codeclimate / ruby-test-reporter

DEPRECATED Uploads Ruby test coverage data to Code Climate
https://codeclimate.com
Other
92 stars 92 forks source link

Heroku CI support #183

Open zharikovpro opened 7 years ago

zharikovpro commented 7 years ago

When I run command locally, it works fine. It works fine on CodeShip, too. That same command fails on Heroku, though.


bundle exec rspec && bundle exec codeclimate-test-reporter

70 examples, 0 failures
Randomized with seed 16670
Coverage report generated for RSpec to /app/coverage. 3377 / 7074 LOC (47.74%) covered.
I, [2017-03-26T15:45:30.729786 #1192]  INFO -- : Reporting coverage data to Code Climate.
fatal: Not a git repository: './.git'
fatal: Not a git repository: './.git'
fatal: Not a git repository: './.git'
bundler: failed to load command: codeclimate-test-reporter (/app/vendor/bundle/ruby/2.3.0/bin/codeclimate-test-reporter)
CodeClimate::TestReporter::InvalidPayload: A git commit timestamp was not found in the test report payload
  /app/vendor/bundle/ruby/2.3.0/gems/codeclimate-test-reporter-1.0.3/lib/code_climate/test_reporter/payload_validator.rb:16:in `validate'
  /app/vendor/bundle/ruby/2.3.0/gems/codeclimate-test-reporter-1.0.3/lib/code_climate/test_reporter/payload_validator.rb:11:in `validate'
  /app/vendor/bundle/ruby/2.3.0/gems/codeclimate-test-reporter-1.0.3/lib/code_climate/test_reporter/formatter.rb:27:in `format'
  /app/vendor/bundle/ruby/2.3.0/gems/codeclimate-test-reporter-1.0.3/lib/code_climate/test_reporter.rb:19:in `run'
  /app/vendor/bundle/ruby/2.3.0/gems/codeclimate-test-reporter-1.0.3/bin/codeclimate-test-reporter:16:in `<top (required)>'
  /app/vendor/bundle/ruby/2.3.0/bin/codeclimate-test-reporter:22:in `load'
  /app/vendor/bundle/ruby/2.3.0/bin/codeclimate-test-reporter:22:in `<top (required)>'
-----> test command `bundle exec rake factory_girl:lint && bundle exec rspec && bundle exec codeclimate-test-reporter` failed with exit status 1```

Any clues on how to fix this?
dblandin commented 7 years ago

Hi @zharikovpro,

When submitting test coverage data to Code Climate, we require a couple pieces of git commit information so that we can reliably link your test coverage report to the appropriate code:

If git is installed and we have access to a git repository, we can grab these values from git itself. Otherwise, we have to rely on environment variables set by the CI provider.

We fetch those environment variables, based on the CI provider, here: https://github.com/codeclimate/ruby-test-reporter/blob/master/lib/code_climate/test_reporter/ci.rb

From the output you shared, it seems like we don't have access to a git repository and we don't currently map values from Heroku CI set environment variables. After briefly looking at the Heroku CI docs I wasn't able to determine if these values are available during a CI run.

If they are, all that should be necessary is adding a new entry to the Ci class.

zharikovpro commented 7 years ago

Will forward this to the Heroku team, they could probably fix it, thanks.

appleton commented 7 years ago

Hello, I'm on the Heroku CI team and I'd love for this to work properly!

There's no git repo present on Heroku CI runs and I don't think we could add that without some significant work but I could expose some more information as environment variables.

We already have HEROKU_TEST_RUN_ID and HEROKU_TEST_RUN_BRANCH, I could add HEROKU_TEST_RUN_COMMIT_SHA if that helps?

dblandin commented 7 years ago

Hey @appleton! Thanks for following up!

A HEROKU_TEST_RUN_COMMIT_SHA environment variable would definitely help!

To completely allow test coverage submission from a Heroku CI build to Code Climate without a git repo, it would also help to have these additional environment variables available:

Name Value
CI_NAME "heroku" or "herokuci"
HEROKU_TEST_RUN_TIMESTAMP Unix timestamp of the build time
HEROKU_TEST_RUN_COMMITTED_AT_TIMESTAMP Unix timestamp of the committed at time

The names can vary but these are the remaining bit of information we would need to generate a valid payload within the test reporter.

Thanks so much for your help! Let me know if you have any further questions.

dwightwatson commented 7 years ago

Any update on this one? The same issue is still present, it would be great to have this working with Heroku CI.

a-b commented 7 years ago

Please let us know ETA for this feature.

a-b commented 7 years ago

@appleton any updates on this issue?

dblandin commented 7 years ago

@a-b I haven't heard anything from the Heroku team. I'll try to get in touch with them via another channel and report back.

Thanks for the nudge!

dblandin commented 7 years ago

@a-b Also, as we're winding down development on this ruby-specific reporter, I'd recommend checking out our new test reporter and migrating to it if you're interested!

https://github.com/codeclimate/test-reporter https://codeclimate.com/changelog/591c99570d51f6481f001291

It's decoupled from the coverage tool used to generate the report, distributed as a static binary, and has support for parallelized builds. 🎉

a-b commented 7 years ago

Thanks for update @dblandin! It would be awesome to include couple words in README on how to configure this new test-reporter to work with Heroku CI for the rails app.

joshwlewis commented 7 years ago

Hey y'all -- Sorry for the delay here. @appleton has been on an extended PTO, so he's not around to respond. However, we did add HEROKU_TEST_RUN_COMMIT_VERSION (which reports recently in response to this (and similar issues).

I'll bring up the other suggestions (CI_NAME, HEROKU_TEST_RUN_TIMESTAMP, and HEROKU_TEST_RUN_COMMITTED_AT_TIMESTAMP) with the team.

dblandin commented 7 years ago

Hey @joshwlewis,

Thank you for the update! Out of those three, HEROKU_TEST_RUN_COMMITTED_AT_TIMESTAMP would definitely be the most important as it's a required value, either from the git repo itself (which is unavailable during Heroku CI runs) or the environment.

If we had that value available, we could address and close this issue easily. Thanks!

dja commented 7 years ago

If we used this solution: https://github.com/codeclimate/test-reporter/issues/226#issuecomment-335218428, what other ENV variables can we send to the ruby test reporter to ensure things run smoothly? (and how do we format them)

dblandin commented 7 years ago

Hey @dja,

That workaround is specific to the new test-reporter. We'd have to support it differently within this reporter.

I'd recommend opening a PR on the new test-reporter repo to support the Heroku env variables that are available and generate the committed at timestamp until Heroku CI supports it.

Here's a PR that recently added Drone CI support: https://github.com/codeclimate/test-reporter/pull/215

dja commented 6 years ago

Hey @joshwlewis, just following up here :) Any progress on making those variables available?

pmaderamitlasf1 commented 6 years ago

@appleton , I am currently using HEROKU CI , unfortunately I can not run functional test which are part of my source code, because the review app url is dynamically changing , is it possible to have HEROKU_APP_NAME/ WEB URL accessible for HEROKU CI example :HEROKU_TEST_RUN_ID is available for my test script. Appreciate you help on this. OR do you suggest any workaround for this?

appleton commented 6 years ago

Hey @pmaderamitlasf1, I don’t work at Heroku any more but I’ll cc @joshwlewis ☝️

pmaderamitlasf1 commented 6 years ago

Thanks @appleton . @joshwlewis can you please help me on this?