unable to differentiate actual tests and fake tests being printed inside tests

[garu]

(originally submitted by @karenetheridge on RT#100337)

CPAN.pm has tests that output "Result: PASS", which are parsed by cpanm-reporter as PASS even if the test itself fails.

$ cpanm --mirror-only --mirror http://mirrors.gossamer-threads.com/CPAN/ CPAN; cpanm-reporter
CPAN
--> Working on CPAN
Fetching http://mirrors.gossamer-threads.com/CPAN/authors/id/A/AN/ANDK/CPAN-2.05.tar.gz ... OK
Configuring CPAN-2.05 ... OK
Building and testing CPAN-2.05 ... FAIL
! Installing CPAN failed. See /Users/ether/.cpanm/work/1415990050.50326/build.log for details. Retry with --force to force install it.
sending: (http://mirrors.gossamer-threads.com/CPAN/authors/id/A/AN/ANDK/CPAN-2.05.tar.gz, ANDK, CPAN-2.05, PASS)

[garu]

There really seems to be no way around this until we are able to get some information from cpanm on what distribution is being tested and what was the output.

[garu]

there is still hope if we manage to tap the actual tests being run. This would allow us to monitor and report independently of cpan clients!

That is the dream (/cc @exodist @leont)

[michal-josef-spacek]

I touch this issue today. The installation directory hasn't rights to install files. cpanm-reporter:

woman:~> cpanm Mo::utils --reinstall
--> Working on Mo::utils
Fetching http://www.cpan.org/authors/id/S/SK/SKIM/Mo-utils-0.27.tar.gz ... OK
Configuring Mo-utils-0.27 ... OK
Building and testing Mo-utils-0.27 ... FAIL
! Installing Mo::utils failed. See /home/skim/.cpanm/work/1716469591.2336625/build.log for details. Retry with --force to force install it.
woman:~> cpanm-reporter
sending: (http://www.cpan.org/authors/id/S/SK/SKIM/Mo-utils-0.27.tar.gz, SKIM, Mo-utils-0.27, PASS)

cat /home/skim/.cpanm/work/1716469591.2336625/build.log:

[SKIP]
t/Mo-utils/12-check_number.t ........... ok
t/Mo-utils/13-check_mumber_min.t ....... ok
t/Mo-utils/14-check_number_of_items.t .. ok
t/Mo-utils/15-check_number_range.t ..... ok
t/Mo-utils/16-check_regexp.t ........... ok
t/Mo-utils/17-check_required.t ......... ok
t/Mo-utils/18-check_string.t ........... ok
t/Mo-utils/19-check_string_begin.t ..... ok
t/Mo-utils/20-check_strings.t .......... ok
All tests successful.
Files=20, Tests=130,  2 wallclock secs ( 0.06 usr  0.03 sys +  1.38 cusr  0.24 csys =  1.71 CPU)
Result: PASS
Manifying 1 pod document
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ERROR: Can't create '/home/skim/perl5/lib/perl5/Mo'
Do not have write permissions on '/home/skim/perl5/lib/perl5/Mo'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 at -e line 1.
make: *** [Makefile:644: pure_site_install] Chyba 13
-> FAIL Installing Mo::utils failed. See /home/skim/.cpanm/work/1716469591.2336625/build.log for details. Retry with --force to force install it.