Why is cpanm-reporter failing to report on this distro?

cpanm-reporter is failing to generate a report for a CPAN distribution which fails to install because two of its prerequisites fail to install. I would think that cpanm-reporter would generate a report in this case, so I'm puzzled as to why it is failing to do so.

My Procedure

I installed Perl 5 blead (commit d2e38af7) for testing purposes and installed cpanm against that perl. (I can supply you with the shell script I use for this purpose upon request.)
For readability, I renamed the directory under which perl was installed to reporter, i.e.:
```
/home/jkeenan/testing/reporter/bin/perl
                               bin/cpanm
                               .cpanm/
                               lib/
```
I then created a symlink named d2e38af7 to reporter. (This was needed so that bin/cpanm did not get confused as to its perl interpreter.)

I created a .cpanreporter directory and set some environmental variables.

$ mkdir .cpanreporter
$ export PERL_CPANM_HOME=/home/jkeenan/testing/reporter/.cpanm
$ export PERL_CPAN_REPORTER_DIR=/home/jkeenan/testing/reporter/.cpanreporter

I installed App::cpanminus::reporter using bin/cpanm.
```
$ ./bin/cpanm App::cpanminus::reporter
```
I setup cpanm-reporter using defaults and copying an already existing metabase_id.json into .cpanreporter/.
```
$ ./bin/cpanm-reporter --setup
```
I used bin/cpanm to install some modules, then used cpanm-reporter to generate and transmit reports. Because my cpanm is not located directly underneath my home directory, I created an alias to get a shorter version of the cpanm-reporter invocation.
```
$ ./bin/cpanm Text::CSV_XS
$ alias cpanm-reporter="./bin/cpanm-reporter --build_dir=$PERL_CPANM_HOME"
$ cpanm-reporter
```

I verified that reports had been generated and transmitted.

$ grep -n 'Text-CSV_XS' .cpanreporter/reports-sent.db
4:test PASS Text-CSV_XS-1.34 (perl-5.27.8) x86_64-linux 4.4.0-104-generic

I went to fast-matrix.cpantesters.org and verified that my report had been received. (My report is the one dated 2017-12-24 15:05.)

From other information I had reason to believe that CPAN distribution Data-Thunk would fail to install against blead (or, for that matter, against perl-5.26) because two of its prerequisites, Data-Swap and UNIVERSAL-ref, would fail to install. I installed some, but not all, of its prerequisites and then tried Data-Thunk.

$ ./bin/cpanm Data::Thunk

Here are excerpts from the build.log for that run. I have trimmed out lines pertaining to modules which installed successfully.

cpanm (App::cpanminus) 1.7043 on perl 5.027008 built for x86_64-linux
Work directory is /home/jkeenan/testing/reporter/.cpanm/work/1514128817.1627
You have make /usr/bin/make
You have LWP 6.31
You have /bin/tar: tar (GNU tar) 1.28
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.
You have /usr/bin/unzip
Searching Data::Thunk () on cpanmetadb ...
--> Working on Data::Thunk
Fetching http://www.cpan.org/authors/id/F/FL/FLORA/Data-Thunk-0.07.tar.gz
-> OK
Unpacking Data-Thunk-0.07.tar.gz
Entering Data-Thunk-0.07
Checking configure dependencies from META.json
Checking if you have ExtUtils::MakeMaker 6.58 ... Yes (7.30)
Configuring Data-Thunk-0.07
Running Makefile.PL
Warning: prerequisite Check::ISA 0 not found.
Warning: prerequisite Data::Swap 0 not found.
Warning: prerequisite Devel::Refcount 0 not found.
Warning: prerequisite UNIVERSAL::ref 0 not found.
Warning: prerequisite namespace::clean 0 not found.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Data::Thunk
Writing MYMETA.yml and MYMETA.json
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have overload 0 ... Yes (1.29)
Checking if you have Carp 0 ... Yes (1.45)
Checking if you have B 0 ... Yes (1.73)
Checking if you have Test::More 0.89 ... Yes (1.302120)
Checking if you have Check::ISA 0 ... No
Checking if you have Devel::Refcount 0 ... No
Checking if you have Data::Swap 0 ... No
Checking if you have Sub::Exporter 0 ... Yes (0.987)
Checking if you have UNIVERSAL::ref 0 ... No
Checking if you have Try::Tiny 0 ... Yes (0.30)
Checking if you have Scalar::Util 0 ... Yes (1.49)
Checking if you have namespace::clean 0 ... No
Checking if you have ok 0 ... Yes (1.302120)
==> Found dependencies: Check::ISA, Devel::Refcount, Data::Swap, UNIVERSAL::ref, namespace::clean
Searching Check::ISA (0) on cpanmetadb ...
--> Working on Check::ISA
Fetching http://www.cpan.org/authors/id/M/MA/MANWAR/Check-ISA-0.09.tar.gz
[snip]
Successfully installed Check-ISA-0.09
[snip]
--> Working on Devel::Refcount
Fetching http://www.cpan.org/authors/id/P/PE/PEVANS/Devel-Refcount-0.10.tar.gz
[snip]
Successfully installed Devel-Refcount-0.10
[snip]
--> Working on Data::Swap
Fetching http://www.cpan.org/authors/id/X/XM/XMATH/Data-Swap-0.08.tar.gz
-> OK
Unpacking Data-Swap-0.08.tar.gz
Entering Data-Swap-0.08
Checking configure dependencies from META.yml
Configuring Data-Swap-0.07
Running Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Data::Swap
Writing MYMETA.yml and MYMETA.json
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have ExtUtils::MakeMaker 0 ... Yes (7.30)
Building and testing Data-Swap-0.07
cp lib/Data/Swap.pm blib/lib/Data/Swap.pm
Running Mkbootstrap for Swap ()
chmod 644 "Swap.bs"
"/home/jkeenan/testing/reporter/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- Swap.bs blib/arch/auto/Data/Swap/Swap.bs 644
"/home/jkeenan/testing/reporter/bin/perl" "/home/jkeenan/testing/d2e38af7/lib/perl5/5.27.8/ExtUtils/xsubpp"  -typemap '/home/jkeenan/testing/d2e38af7/lib/perl5/5.27.8/ExtUtils/typemap'  Swap.xs > Swap.xsc
mv Swap.xsc Swap.c
cc -c   -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2   -DVERSION=\"0.08\" -DXS_VERSION=\"0.08\" -fPIC "-I/home/jkeenan/testing/d2e38af7/lib/perl5/5.27.8/x86_64-linux/CORE"   Swap.c
rm -f blib/arch/auto/Data/Swap/Swap.so
cc  -shared -O2 -L/usr/local/lib -fstack-protector-strong  Swap.o  -o blib/arch/auto/Data/Swap/Swap.so  \
      \

chmod 755 blib/arch/auto/Data/Swap/Swap.so
"/home/jkeenan/testing/reporter/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- Swap.bs blib/arch/auto/Data/Swap/Swap.bs 644
PERL_DL_NONLAZY=1 "/home/jkeenan/testing/reporter/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
#     Failed test (t/01_deref.t at line 62)
#                   undef
#     doesn't match '(?^:^Tie::StdArray=ARRAY\(0x[0-9A-Fa-f]+\)\z)'
# Looks like you failed 1 tests of 13.
t/01_deref.t ..
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/13 subtests
t/02_swap.t ... ok

Test Summary Report
-------------------
t/01_deref.t (Wstat: 256 Tests: 13 Failed: 1)
  Failed test:  13
  Non-zero exit status: 1
Files=2, Tests=33,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.04 cusr  0.00 csys =  0.06 CPU)
Result: FAIL
Failed 1/2 test programs. 1/33 subtests failed.
Makefile:949: recipe for target 'test_dynamic' failed
make: *** [test_dynamic] Error 255
-> FAIL Installing Data::Swap failed. See /home/jkeenan/testing/reporter/.cpanm/work/1514128817.1627/build.log for details. Retry with --force to force install it.
Searching UNIVERSAL::ref (0) on cpanmetadb ...
--> Working on UNIVERSAL::ref
Fetching http://www.cpan.org/authors/id/J/JJ/JJORE/UNIVERSAL-ref-0.14.tar.gz
-> OK
Unpacking UNIVERSAL-ref-0.14.tar.gz
Entering UNIVERSAL-ref-0.14
Checking configure dependencies from META.yml
Checking if you have ExtUtils::MakeMaker 6.58 ... Yes (7.30)
Configuring UNIVERSAL-ref-0.14
Running Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for UNIVERSAL::ref
Writing MYMETA.yml and MYMETA.json
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have B 0 ... Yes (1.73)
Checking if you have B::Utils 0 ... Yes (0.27)
Checking if you have Test::More 0 ... Yes (1.302120)
Checking if you have overload 0 ... Yes (1.29)
Checking if you have XSLoader 0 ... Yes (0.29)
Building and testing UNIVERSAL-ref-0.14
cp lib/UNIVERSAL/ref.pm blib/lib/UNIVERSAL/ref.pm
Running Mkbootstrap for ref ()
chmod 644 "ref.bs"
"/home/jkeenan/testing/reporter/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- ref.bs blib/arch/auto/UNIVERSAL/ref/ref.bs 644
"/home/jkeenan/testing/reporter/bin/perl" "/home/jkeenan/testing/d2e38af7/lib/perl5/5.27.8/ExtUtils/xsubpp"  -typemap '/home/jkeenan/testing/d2e38af7/lib/perl5/5.27.8/ExtUtils/typemap' -typemap '/home/jkeenan/testing/reporter/.cpanm/work/1514128817.1627/UNIVERSAL-ref-0.14/typemap'  ref.xs > ref.xsc
mv ref.xsc ref.c
cc -c   -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2   -DVERSION=\"0.14\" -DXS_VERSION=\"0.14\" -fPIC "-I/home/jkeenan/testing/d2e38af7/lib/perl5/5.27.8/x86_64-linux/CORE"   ref.c
ref.xs: In function ‘universal_ref_fixupop’:
ref.xs:92:9: error: ‘OP {aka struct op}’ has no member named ‘op_sibling’
   if ( o->op_sibling ) {
         ^
ref.xs:94:28: error: ‘OP {aka struct op}’ has no member named ‘op_sibling’
     universal_ref_fixupop(o->op_sibling);
                            ^
Makefile:333: recipe for target 'ref.o' failed
make: *** [ref.o] Error 1
-> FAIL Installing UNIVERSAL::ref failed. See /home/jkeenan/testing/reporter/.cpanm/work/1514128817.1627/build.log for details. Retry with --force to force install it.
Searching namespace::clean (0) on cpanmetadb ...
--> Working on namespace::clean
Fetching http://www.cpan.org/authors/id/R/RI/RIBASUSHI/namespace-clean-0.27.tar.gz
[snip]
Successfully installed namespace-clean-0.27
[snip]
-> FAIL Installing the dependencies failed: Module 'UNIVERSAL::ref' is not installed, Module 'Data::Swap' is not installed
-> FAIL Bailing out the installation for Data-Thunk-0.07.
13 distributions installed

The failures in Data-Swap and UNIVERSAL-ref first appeared in the 5.25.* development cycle and have not been corrected to date. Note that Data-Swap failed during make test while UNIVERSAL-ref failed during make.

I then ran cpanm-reporter again and inspected the results.
```
$ cpanm-reporter
$ $ ack '(Data-Swap|UNIVERSAL-ref|Data-Thunk)'   .cpanreporter/reports-sent.db
test NA Data-Swap-0.08 (perl-5.27.8) x86_64-linux 4.4.0-104-generic
test FAIL UNIVERSAL-ref-0.14 (perl-5.27.8) x86_64-linux 4.4.0-104-generic
```
Note the following:
- 1 Data-Swap was graded NA by cpanm-reporter despite being graded FAIL by the harness.
- 2 UNIVERSAL-ref was graded FAIL.
- 3 Most importantly, no report was generated for Data-Thunk, even though that was the only module specifically passed to bin/cpanm.
I somewhat understand items 1 and 2 but was very surprised by item 3.

Thoughts

I would have expected a report to be generated for Data-Thunk, even if that report only said something to the effect that its prerequisites were not satisfied.

Now, I recognize that this is not just a problem with cpanm-reporter. If you go to the matrix, you see that there have been no reports generated for Data-Thunk later than perl-5.25.1 -- not even NA or UNKNOWN reports. I suspect that most CPANtesters reports are filed with programs based on cpan rather than cpanm. So there may be a problem with the way that all the various reporter programs are parsing build logs -- cpan build logs as well as cpanm build logs.

Summary

Do you have any insight as to why such reports are not being generated?

Feature or bug?

Is there anything we can do to rectify the situation?

Thank you very much. Jim Keenan

garu / App-cpanminus-reporter