Hubot should use the repo status API

winhamwr commented 11 years ago

jenkins-pstat let's you request Jenkins builds, whose build page will be posted to the pull request. Instead of posting messages, uses the github repository status API to mark the pull request. Also extend this functionality by making Hubot follow up and check the status of those builds.

Note: It might be better to split this in to several pull requests. A suggestion would be:

Just post a simple message on the pull request on failure with a link to the build
Use the failure message with all of the nice details
Replace the build started message with the github repo status API calls
Add repo status API calls for success/failure
Repo statuses on the pull request

Build started, but not finished

Instead of just posting a message with the link to the build, use the github repo status API.

API Arguments:

{
  "state": "pending",
  "target_url": "http://jenkins.pstattest.com/job/pstat_ticket/759/",
  "description": "#759_issue_1072 is running."
}

All downstream jobs succeed

API Arguments:

{
  "state": "success",
  "target_url": "http://jenkins.pstattest.com/job/pstat_ticket/759/",
  "description": "#759_issue_1072 succeeded! 6 downstream projects completed successfully."
}

Any downstream jobs fail

API Arguments:

{
  "state": "failure",
  "target_url": "http://jenkins.pstattest.com/job/pstat_ticket/759/",
  "description": "#759_issue_1072 failed! 2 downstream jobs failed."
}

Then, also post a comment on the pull requests like:

Build [#749_issue_1072]() failed. [1 of 212 tests](http://jenkins.pstattest.com/job/pstat_ticket/758/aggregatedTestReport/) failed.

The following jobs failed:
* [pstat_ticket_selenium_2](http://jenkins.pstattest.com/job/pstat_ticket_selenium_2/701/)

You can get the number of tests via: http://jenkins.pstattest.com/job/pstat_ticket/758/aggregatedTestReport/api/json

Performing follow-up with Hubot to check build status

There are two possible ways of doing this. You can either have Jenkins notify Hubot when things are done (preferable) or have Hubot periodically poll for results.

Hubot polling

Use the redis-brain to add persistence and hubot-cron to fire off requests for all of the jobs who have been started but not yet finished.

Jenkins notifications

Use the Jenkins Notification Plugin to fire off status notifications to a Hubot API endpoint that you build. That endpoint will interpret the status notifications and perform the appropriate action.

stemchan commented 11 years ago

I was looking into using http://jenkins.pstattest.com/job/pstat_ticket/758/aggregatedTestReport/api/json to get the failed number of tests as well but the aggregated test report only reflects the Selenium tests and not the unit tests. For example, this build has 5 failed nodes but only 2 reported test failures in the aggregated test report.

You also have the following part for the pull request comment:

The following jobs failed:
* [pstat_ticket_selenium_2](http://jenkins.pstattest.com/job/pstat_ticket_selenium_2/701/)

I was having trouble finding an API endpoint that lets me find all the downstream builds associated to a pstat_ticket build.

stemchan commented 11 years ago

This pull request is a variation of some of the things mentioned in the spec. It uses Hubot's HTTP listener to serve as an endpoint for the Jenkins Notification plugin.

Whenever Hubot receives a notification that a build has finished, it stores that data using redis-brain. I set up a Redis To Go Nano plan to our existing Heroku app for this, which has 5 MB of space. Since it's only serving as a temporary key value store for the duration of a build, we'll probably never need any more than that.

I'd like to eventually have a more detailed report of the number of tests that failed and maybe even the actual names of the failing tests, but that should probably be its own ticket.

kylegibson commented 11 years ago

We should pair on adding some documentation to explain how this works and how to configure it.

stemchan commented 11 years ago

@kylegibson I changed the way the brain is storing the data like you had suggested and added the env vars. Whenever you want to pair on the documentation, let me know.

PolicyStat / hubot