search

PDLPorters / pdl

Scientific computing with Perl
http://pdl.perl.org
Other
90 stars 46 forks source link

SF#334 Test coverage is incomplete #34

Open perldl-bot opened 9 years ago

perldl-bot commented 9 years ago

From http://sourceforge.net/p/pdl/bugs/334

duffee commented 5 days ago

I started having a look at this one by running Devel::Cover across PDL with

HARNESS_PERL_SWITCHES=-MDevel::Cover make test
cover

(it took 23 hours on my machine) It reports 4% coverage for blib-lib-PDL-IO-FITS-pm.html, hitting only the BEGIN statements. The tests in IO/FITS/t/fits.t exercise a lot more than that, so something is preventing it from running correctly. I can't figure out how to get Devel::Cover to Do What I Mean, not helped by PDL's archaic test layout. I've been to the coveralls site and still can't pick out any hints on how to run coverage on my local machine.

Is trying to generate a coverage report doomed to failure or does someone have a suggestion?

mohawk2 commented 5 days ago

Thank you for wanting to contribute to this important issue! Ironically, I've been focusing on it myself recently, with getting Test::PDL ready to incorporate into main-PDL.

We run coverage tests in our CI on every commit (it's the Coveralls tick or cross). It isn't very quick, but where CI of a fairly complete build (with many or all of the optional builds present) takes about 6 mins (less when ccache is helping), with coverage it's about 8 (less with ccache). I don't know why it would take 23 hours. The canonical way to run coverage tests is cover -test, though your incantation above should work similarly. Take a look at .github/workflows/ci.yml and https://github.com/PDLPorters/devops/blob/master/github-actions/ci-dist/action.yml (the "with coverage" step). It's doing $MYPERL -S cover -test -relative_only -gcov_chdir -report Coveralls.

You allude to an "archaic" layout of tests. Before I changed it, nearly all the tests were in the top t directory, and needed various complicated guard mechanisms to make themselves not run if their component (being optional) wasn't built. That adds considerably to the difficulty of splitting up the unwieldy blob that PDL still is, but soon won't be. It also added to the burden of maintaining the complicated guard mechanisms. Moving the tests next to their relevant code is more modular.

duffee commented 4 days ago

Upgrading Devel::Cover from old system version 1.36 to new cpan version 1.44 (and using your CI step) made the cover run in a decent time. (23 hours was a sign that I was doing something wrong, but didn't know what)

It reports 4% which is still low if my eyeballs can estimate how much fits.t exercises, but it will have to wait until next week for me to look at it.

I think this is now the 28th anniversary of PDL and I assumed that the tests buried in subdirectories were an artifact of it growing organically before CPAN practices converged. If I have any decent ideas on how to handle the optional components, I'll bring it up for discussion.