ci-reporter / ci_reporter_minitest

Connects CI::Reporter to Minitest
MIT License
6 stars 7 forks source link

Reports directory or XML files are not created #7

Open sipani909 opened 3 years ago

sipani909 commented 3 years ago

I have added the following rake code:

require 'digest' require 'ci/reporter/rake/minitest' ENV["CI_REPORTS"] = "reports/" . . . . desc 'Run Subset of Tests' task :subset => 'ci:setup:minitest' do . . .

But no reports folder or xml files are being generated.

ruby 2.6.3p62 (2019-04-16 revision 67580) Minitest-5.14.4 ci_reporter_minitest-1.0.0

sethpeters940 commented 3 years ago

I'm having the same problem as @sipani909 on ruby 3.0.0 and ci_reporter_minitest-1.0.0.

hakanai commented 1 year ago

Same here on ruby 3.2.0 but I'm not sure whether it's the Ruby upgrade or the Rails upgrade which did it, because we went through major versions of both in the process and didn't notice reports were missing until the very end.

I've done some investigation. Minitest does its best to prevent being investigated by setting reporter to nil as soon as it gets a reference for itself, so I had to jump through some hoops, writing my own plugin to steal the reporter at startup, and then looking at what reporters had been added to it by the time the test ran.

I see three reporters on it:

So the reporter seems to be present, but since this SuppressedSummaryReporter appears to be so broken that it can't print itself out as a string, perhaps it's also breaking the report chain somehow - is it possible getting one error in a minitest reporter will cause it to skip all subsequent reporters?

Minitest::CompositeReporter's implementation would appear to back this theory up:

https://github.com/minitest/minitest/blob/master/lib/minitest.rb#L881-L885

hakanai commented 1 year ago

It looks like there are at least two issues at play - the issue in minitest where some other plugin I had was throwing an error that stopped this plugin being run, but after removing the other plugin, ci_reporter also throws an error. Ironically for the exact same reason.

The issue with ci_reporter is already filed: https://github.com/ci-reporter/ci_reporter/issues/167

The fix is also already committed, but there hasn't been a release in forever.