search

bugcrowd / test-summary-buildkite-plugin

Buildkite plugin to summarise all test failures as an annotation
MIT License
68 stars 39 forks source link

Usage feedback #4

Closed micheletest closed 6 years ago

micheletest commented 6 years ago

I tried out this plugin today and it worked great. Thank you for developing it.

Background: I was going to use the buildkite provided plugin, but really like the feature to separate my types of tests (as mentioned on the buildkite plugin's issue board). Plus, it seems that this may become the official plugin.

One issue I ran into, which I think is actually a non-issue:

$ buildkite-agent artifact download micheletest/be-results.xml tmp/test-summary
--
  | 2018-04-30 18:02:53 INFO   Searching for artifacts: "micheletest/be-results.xml"
  | 2018-04-30 18:03:53 FATAL  Failed to download artifacts: Get https://agent.buildkite.com/v3/builds/89bc3a56-16b9-4da7-872a-3d92165371a1/artifacts/search?query=micheletest%2Fbe-results.xml: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  | /usr/src/app/lib/test_summary_buildkite_plugin/agent.rb:27:in `run': Command 'buildkite-agent artifact download micheletest/be-results.xml tmp/test-summary' failed (exit status: 1) (RuntimeError)
  | from /usr/src/app/lib/test_summary_buildkite_plugin/input.rb:39:in `files'
  | from /usr/src/app/lib/test_summary_buildkite_plugin/input.rb:77:in `failures_raw'
  | from /usr/src/app/lib/test_summary_buildkite_plugin/input.rb:28:in `failures'
  | from /usr/src/app/lib/test_summary_buildkite_plugin/formatter.rb:15:in `markdown'
  | from /usr/src/app/lib/test_summary_buildkite_plugin/runner.rb:12:in `block in run'
  | from /usr/src/app/lib/test_summary_buildkite_plugin/runner.rb:12:in `map'
  | from /usr/src/app/lib/test_summary_buildkite_plugin/runner.rb:12:in `run'
  | from /usr/src/app/lib/test_summary_buildkite_plugin.rb:17:in `run'
  | from ./bin/run:10:in `<main>'
  |  
  | 🚨 Error: The plugin github.com/bugcrowd/test-summary-buildkite-plugin#v1.0.0 command hook exited with status 1

I fixed that by properly naming my files. I had 15 parallel generated files with the same name of be-results.xml, and intended to rename them to include BUILDKITE_JOB_ID regardless. Once I did that, the download error went away.

tessereth commented 6 years ago

Thanks for letting me know how it went and I'm glad it worked! And hey, you got to actually test that error path. I never actually made the agent fail so good to know the plugin behaves in a vaguely sane manner. I guess buildkite-agent doesn't handle multiple artefacts with the same name nicely? Not a whole lot I can do about that but it's good to know.

toolmantim commented 6 years ago

Strange error there! Sorry you ran into that @micheletest. I think it actually might have been related to a performance slowdown on our artifact "search" download endpoint, making the agent timeout the request. But we've rolled out some fixes.

The agent can definitely handle multiple artifacts with the same name… you'll only end up with a single artifact downloaded though (the last one, unless you use the --step argument to specify exactly which step's to use).

micheletest commented 6 years ago

Thanks @toolmantim, that's interesting that I may have hit upon a performance slowdown. Glad it helped you to roll out some fixes. All my artifact names are unique now, which seems necessary for this reporting anyway. There is no issue here, I'm going to close this.