search

travis-ci / travis-ci

Free continuous integration platform for GitHub projects.
https://travis-ci.org
8.41k stars 724 forks source link

Handle Ant's JUnit XML formatted output from test/spec runs #239

Closed ndbroadbent closed 9 years ago

ndbroadbent commented 12 years ago

Ant's JUnit XML format is a standard output for almost every test framework, and can be read by many continuous integration systems, such as Hudson or Atlassian Bamboo:

We could start by handling the ci_reporter gem, since a lot of projects on Travis CI are Ruby / Rails. ci_reporter provides formatters for Test::Unit, RSpec and Cucumber. If we assume that some projects will be configured to use ci_reporter, then we could just test for the presence of the XML output at the end of the test run, and handle it if it exists. Alternatively, we could detect the presence of a ruby test framework and inject ci_reporter into the project's Gemfile before running the test suite. This idea is possibly related to the issue about generating a Gemfile, since they will both automatically alter the project's environment.

This would allow us to store / notify which tests passed / failed, and include backtraces for each error, etc.

joshk commented 12 years ago

This is an interesting idea, not keen about injecting any gem deps, but we should do some form of smart parsing ourselves with the saved log output and maybe show it in the ui and in the emails.

Is there a way we can use ci reporter ourselves with saved log output?

Sent from my Sega Master System

On 5/09/2011, at 1:31 PM, ndbroadbentreply@reply.github.com wrote:

This is probably on the roadmap, but I haven't seen it mentioned anywhere.

The ci_reporter gem generates XML reports of your test and/or spec runs. Travis CI could then parse the generated XML, and send the formatted failure info (backtrace, line number, etc.) along with the notification email. Could also show the test details in the UI somewhere.

It could be an optional flag in .travis.yml to state that the project already uses ci_reporter. OR, we could do something fancy and inject ci_reporter before the test suite is run.

I think features like this would really start to make Travis CI on par with Hudson / CruiseControl / Bamboo. (is that the aim?)

Reply to this email directly or view it on GitHub: https://github.com/travis-ci/travis-ci/issues/239

ndbroadbent commented 12 years ago

The idea comes from Heroku, who do code injection for Rails apps.

One example of this is rails_log_stdout, a plugin injected into Rails apps to cause their logs to go to stdout, as is required for routing logs into the logging system.

There's probably a heroku dev out there who could give us some pointers, although it doesn't seem like vendoring an extra plugin would be too dangerous.

ci_reporter extends RSpec with a new formatter, so as far as I'm aware it can't be used to post-process the saved log output.

joshk commented 12 years ago

Although heroku does it, I'm not sure we should.

Since it's ruby and rspec specific, it seems like an inflexible solution for other test frameworks and langs.

I'm not adverse to something like this, but I would prefer a log file parser which we can extend for other langs and test frameworks.

Maybe we can talk about it further when I am back from holiday, typing on an iPhone is a pain :)

Sent from my Sega Master System

On 5/09/2011, at 2:56 PM, ndbroadbentreply@reply.github.com wrote:

The idea comes from Heroku, who do code injection for Rails apps.

One example of this is rails_log_stdout, a plugin injected into Rails apps to cause their logs to go to stdout, as is required for routing logs into the logging system.

There's probably a heroku dev out there who could give us some pointers, although it doesn't seem like vendoring an extra plugin would be too dangerous.

ci_reporter extends RSpec with a new formatter, so as far as I'm aware it can't be used to post-process the saved log output.

Reply to this email directly or view it on GitHub: https://github.com/travis-ci/travis-ci/issues/239#issuecomment-1999054

ndbroadbent commented 12 years ago

Sorry if it was misleading to only mention RSpec, but ci_reporter handles Test::Unit, RSpec and Cucumber. It produces XML output that can be read by any continuous integration system that understands Ant’s JUnit report XML format, such as Hudson or Atlassian Bamboo.

Ant's JUnit XML format is a standard output for almost every test framework:

So I guess this issue is really about handling that XML format, and the ci_reporter injection might be a good first step in that direction. While parsing log files is possible, I don't think it's the ideal solution although it could be a fall-back.

Alright, well thanks for your response, and I'm happy to leave it there unless someone else wants to join the discussion. I'll also update this issue with all of this information.

Have a good holiday!

joshk commented 12 years ago

Ok, you sold me, we should get this to work somehow.

Let's talk when I get back and see how to get this to work in a clean way.

Thanks a bundle,

J

Sent from my Sega Master System

On 5/09/2011, at 5:03 PM, ndbroadbentreply@reply.github.com wrote:

Sorry if it was misleading to only mention RSpec, but ci_reporter handles Test::Unit, RSpec and Cucumber. It produces XML output that can be read by any continuous integration system that understands Ant’s JUnit report XML format, such as Hudson or Atlassian Bamboo.

Ant's JUnit XML format is a standard output for almost every test framework:

So I guess this issue is really about handling that XML format, and the ci_reporter injection might be a good first step in that direction. While parsing log files is possible, I don't think it's the ideal solution although it could be a fall-back.

Alright, well thanks for your response, and I'm happy to leave it there unless someone else wants to join the discussion. I'll also update this issue with all of this information.

Have a good holiday!

Reply to this email directly or view it on GitHub: https://github.com/travis-ci/travis-ci/issues/239#issuecomment-2001660

svenfuchs commented 12 years ago

I have a couple of questions about this.

ndbroadbent commented 12 years ago

Hi, happy to answer those questions:

The issue is about obtaining more fine-grained information about the build, instead of just a 0 or 1 exit status. It's great that Travis CI can email me to tell me that something broken. But it would be fantastic if the notification email and the UI could tell me that "these tests have broken in these files, and at these line numbers. Here are the colorized backtraces."

ndbroadbent commented 12 years ago

This is an example of the kind of interface that we could offer (screenshot from Hudson):

Hudson example

joshk commented 12 years ago

REALLY LOVE THIS!!! :)

ndbroadbent commented 12 years ago

I'm really stoked to see the future plans for Travis CI on https://love.travis-ci.org/! I saw the 'Build artifacts' section under 'Future Plans', and was reminded of this feature.

I still really think that it would be awesome if Travis CI could parse the formatted build output, and be able to report that "3 tests failed from the RSpec test suite".

It would still be up to the developers to ensure that the test results are formatted correctly and saved in the right place. Travis could detect the presence of these formatted test results and parse them. This would result in two levels of reporting: Pass/Fail (which we have now), plus more in-depth details (if available), such as which tests were fixed and which were broken.

I need to catch up on the changes to the architecture, but I'd love to discuss the implementation further and help to build this :)

joshk commented 12 years ago

Hey @ndbroadbent,

It is great to hear that you are excited to get involved with this feature!

I do think this feature could grow into a monster, so many we should start planning out some simple first steps.

The simpler the better!

Let me know what you think,

Josh

michaelklishin commented 12 years ago

First question we need to answer is what application this logic should go, worker or Hub or even something new.

ndbroadbent commented 12 years ago

Parsing and storing the test results will only be a small part of the work. Travis CI will be affected in many different places:

So, I think it would be best for the test results to be sent from the Worker, and processed in Travis Hub. That processing code could live in a separate library. The test results would need to be processed before delivering notifications, because we'd want notifications to contain that information. (e.g. "5 tests failed").

This means that the build log would still be displayed in the browser while the build is running, but after the build has finished, more fine-grained information about the test suite is notified and displayed.

The following paragraph also seems to be very relevant to this discussion:

It is fair to say that Travis Hub is the most complex of all Travis CI applications we have today. This is why we carefully evaluate what features are worth adding to the Hub: many features simply won’t be worth the complexity.

I would say that a CI system is mainly about tests, so I think it makes sense for us to provide more information than just pass / fail.

ndbroadbent commented 12 years ago

Note: The following comment is probably way out of scope for now, and is probably the 'monster' you were talking about...

In addition to parsing the XML build results at the end of each build, we could also provide developers with test formatters, which would allow the Travis Worker to parse the stdout stream on the fly. Travis Worker could detect that the stream is parseable (perhaps by a known header and footer), and then send these results to Travis Hub as a test results chunk message (instead of build log chunk). We could start small, by only supporting Test::Unit and RSpec.

joshk commented 12 years ago

I think we need to make this even simpler for now, for example, instead of sending extra info in the email lets add that feature later.

Also, instead of parsing the XML or related output, lets just have it either displayable in its raw format or have it downloadable.

Try to break this down into smaller simpler features which we can build off and enhance.

michaelklishin commented 12 years ago

@ndbroadbent keep in mind that workers stream the output more or less as it comes in, this is why near real-time UI is possible. So parsing XML in the worker is not an option (making workers too aware of what they stream and how that data is used is probably a bad idea). Adding this kind of functionality (build log analysis) to the Hub is OK (especially if this can be isolated enough to be an internal library) but this needs to be done carefully.

I agree with what @joshk says. Not sure what's the minimum viable feature set here, maybe we can discuss this first?

ndbroadbent commented 12 years ago

I also think that's a good idea. In fact, I think that's how our CI system operates (Atlassian Bamboo). IIRC, it treats the XML file as a build artifact, and processes the test results after retrieving it.

Alright, then let's come back to this issue when Travis supports build artifacts. :)

michaelklishin commented 12 years ago

@ndbroadbent please try putting together a minimum viable feature design. This issue is very much related to build artifacts and there are many open questions right now. Coming up with a small but already useful solution will help pushing both features (and probably some more related ones).

It is not something we need to come up with the next week but good ideas often come after you spend some time thinking about something :)

joshk commented 12 years ago

@ndbroadbent Just confirming though, are you interested in helping us with Build Artifacts?

ndbroadbent commented 12 years ago

@michaelklishin - Sure, I will look into it.

@joshk - Yes, very interested in helping. Is there something I can read to learn about the current plans & discussion? Or just hang out on IRC more often?

michaelklishin commented 12 years ago

Hanging around the irc is the best way.

MK

El 10/feb/2012, a las 14:01, Nathan Breply@reply.github.com escribió:

Is there something I should read, to find out more about the current plans & discussion? Or just hang out on IRC more often?

ndbroadbent commented 12 years ago

Hey, sorry guys, I'm going to have to back down from my offer to help with the build artifacts :( It feels like it's a bit too much for me to take on at the moment, but I'd still like to contribute to Travis from time to time. I'm also just about to switch from full-time work to freelancing, so I might have some more free time to donate in the future!

Sorry again for not following up on this sooner, and all the best with your work on Travis CI pro!

svenfuchs commented 12 years ago

No worries, Nathan. That's how it works with OSS projects.

Thanks for letting us know!

On Mar 30, 2012, at 6:41 PM, Nathan B wrote:

Hey, sorry guys, I'm going to have to back down from my offer to help with the build artifacts :( It feels like it's a bit too much for me to take on at the moment, but I'd still like to contribute to Travis from time to time. I'm also just about to switch from full-time work to freelancing, so I might have some more free time to donate in the future!

Sorry again for not following up on this sooner, and all the best with your work on Travis CI pro!

Reply to this email directly or view it on GitHub: https://github.com/travis-ci/travis-ci/issues/239#issuecomment-4846330

DracoBlue commented 12 years ago

+1! Would be really cool to have this feature!

acertain commented 11 years ago

Would be wonderful to have this, another idea is to display a grid of tests (by job or time) and color the cells based on the test status. Also python's nose has junit support

jelder commented 10 years ago

+1

Reading the logs in Travis is a very unpleasant experience. Given the choice, I'm sure most people would rather consume a semantic, structured test result like would be provided by Travis' consumption of JUnit XML.

dexterhu commented 10 years ago

+1 We need test report visualization somewhere online for each test run.

joshk commented 10 years ago

We have something in the works for this, but I don't have an ETA sorry.

bkimminich commented 9 years ago

ping :+1:

alexkalderimis commented 9 years ago

I would be happy to help with this if there is still interest. We have our own system, but it is hacky.

marscher commented 9 years ago

this would be an extremely useful feature. E.g Appveyor provides an API to upload xunit.xml (.net testing output xml) and displays it nicely on a results page. Directly seeing which tests brake eg. for a pull request is a killer feature for maintainers, as they could give direct advice to commiters, what might be wrong.

dguido commented 9 years ago

:+1: I would pay for this feature.

bkimminich commented 9 years ago

I will pay for this feature: https://www.bountysource.com/issues/848108-handle-ant-s-junit-xml-formatted-output-from-test-spec-runs

flozano commented 9 years ago

:+1:

roidrage commented 9 years ago

Closing this issue for now, as we have no immediate plans to add this feature.

Should we add it to the roadmap eventually, we'll make sure to update this ticket.