Open Tolsto opened 2 years ago
We can use rspec_junit_formatter
gem to generate an XML report with tests results. We would need something to convert the XML report to a nice markdown summary.
Hey - I've done exactly what you're looking for with this code:
This assumes rspec running with --json
output (rather than junit, although you could do something similar with junit), written to the spec/reports/rspec*.json
directory. It works well for us in conjunction with the github artifacts uploader/downloader collecting the json formatted spec result from each node:
- name: Upload json rspec reports
uses: actions/upload-artifact@v3
if: always()
continue-on-error: true
with:
name: rspec-json-reports
retention-days: 10
path: |
/your/app/spec/reports/*.json
when rspec is run like:
bundle exec rspec --format json --out spec/reports/rspec_<RSPEC_NODE_NUMBER>.json
# frozen_string_literal: true
require_relative "rspec_report_merger"
require "terminal-table"
require "json"
class GithubActionsStepSummaryCreator
def initialize(overall_summary = RSpecReportMerger.new.overall_summary,
per_dir_summary = RSpecReportMerger.new.per_dir_summary)
@overall_summary = overall_summary
@per_dir_summary = per_dir_summary
@spec_reports = Dir["spec/reports/rspec*.json"]
raise "Error: No spec reports discovered!" if @spec_reports.empty?
end
def headings
["directory"] + @overall_summary.keys
end
def main_app_suite_data
[["**All tests**"] + @overall_summary.values.map{ |v| v.round(2) }]
end
def per_dir_rows
@per_dir_summary.map do |dir_spec|
[
dir_spec.first,
dir_spec.last["duration"],
dir_spec.last["example_count"],
dir_spec.last["failure_count"],
dir_spec.last["pending_count"],
]
end
end
def add_step_summary
rows = main_app_suite_data + per_dir_rows
rows.sort_by!{ |data| -data[1] } # sort by duration
table = Terminal::Table.new(
headings: headings,
rows: rows,
style: { border: :markdown }
)
puts table
message = "### Spec report: :rocket: \n \n #{table}"
File.write(ENV["GITHUB_STEP_SUMMARY"] || "/tmp/summary", message, mode: "a+")
end
def run
add_step_summary
end
end
GithubActionsStepSummaryCreator.new.run
# frozen_string_literal: true
class RSpecReportMerger
def initialize(options = {})
@spec_reports = Dir["spec/reports/rspec*.json"]
@options = options
raise "Error: No spec reports discovered!" if @spec_reports.empty?
end
def read_reports(key)
@spec_reports.flat_map do |file|
JSON.parse(File.read(file)).fetch(key)
end
end
def per_dir_summary
# examples grouped by path
grouped = read_reports("examples").group_by do |example|
example["file_path"].split("/").first(3).last(2)
end
result = {}
grouped.each_key do |key|
duration = grouped[key].map{ |ex| ex["run_time"] }.sum
status_counts = grouped[key].map{ |ex| ex["status"] }.tally
result[key.join("/")] = {
"example_count" => grouped[key].count,
"failure_count" => status_counts["failed"] || 0,
"pending_count" => status_counts["pending"] || 0,
"duration" => (duration/60).round(2)
}
end
result
end
def overall_summary
result = {
"duration" => 0,
"example_count" => 0,
"failure_count" => 0,
"pending_count" => 0,
}
summaries = read_reports("summary")
summaries.each do |summary|
result["duration"] += summary["duration"]
result["example_count"] += summary["example_count"]
result["failure_count"] += summary["failure_count"]
result["pending_count"] += summary["pending_count"]
end
result
end
end
The code is quick and dirty and could definitely be improved, but it works. Here's what it looks like:
Note if you're missing rspec data and are running in queue mode, you might need to follow these instructions: https://docs.knapsackpro.com/ruby/rspec/#queue-mode
@singhprd Thank you for sharing this example! 🚀
Here are ideas what to do before we close this issue. Info for Knapsack team:
Maybe we could add an example of using the above script and GitHub Actions to our Rails example project. Or we could create an article on our blog.
In the case of Queue Mode, we could generate JSON reports this way.
We could then update cookbook docs and add the example config that uses GitHub Actions job summaries.
In the installation guide, we could let users know that they can check the cookbook if they want.
Github Actions has a new features that allows to add arbitrary markdown code for job summaries. It would be great if Knapsack Pro would support that out of the box when running on Github.
https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/
This is a library for dotnet that has already support for summaries: https://github.com/Tyrrrz/GitHubActionsTestLogger