karenetheridge
commented
5 years ago If xt tests don't run during release, when would they run? You are the author and releaser, so surely you'd want to know about failures at release time.
Open wollmers opened 5 years ago
karenetheridge
commented
5 years ago If xt tests don't run during release, when would they run? You are the author and releaser, so surely you'd want to know about failures at release time.
karenetheridge
commented
5 years ago xt/author-pod-coverage.t ...... skipped: these tests are for author testing
This one is surprising to me -- 'minil release' should be setting AUTHOR_TESTING=1 so this test can run.
wollmers
commented
5 years ago @karenetheridge According to which spec should tests under xt run? Or, opposite question, where should I place tests which need external huge data (here GoldStandard), compare each iteration with the result of the original version, do extensive benchmarks--both speed and f-score--etc., generate very huge permutions of combinations? Really, should they be performed during build and release of the package? They are module-specific development tools, to be used by developers. They are not targeting the average user, thus not scripts/ or examples/.
skaji
commented
5 years ago I think it is reasonable that Minilla runs test in xt.
If you don't want Minilla to run some tests,
then guard them by skip_all() function or locate them in other directories.
karenetheridge
commented
5 years ago @wollmers: this is covered in the Lancaster Consensus document: https://github.com/Perl-Toolchain-Gang/toolchain-site/blob/master/lancaster-consensus.md#environment-variables-for-testing-contexts (the placement of files into xt/ is not explicitly covered, but convention is that tests there are in subdirs named similarly for the environment variables that would protect them, e.g. xt/author/, xt/release/ etc.)
wollmers
commented
5 years ago milla/dzil doesn't run every xt/*.t, so something must be wrong: the spec, milla/dzil or minil. Of course I could (and should) skip them.
Just to be sure, that milla/dzil do not run them and minil does:
helmut@Helmuts-MacBook-Pro:~/github/perl/LCS-BV$ time minil test --release
Creating working directory: /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5
cp Build.PL /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/Build.PL
cp Changes /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/Changes
cp LICENSE /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/LICENSE
cp META.json /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/META.json
cp README.md /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/README.md
cp cpanfile /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/cpanfile
cp dist.ini /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/dist.ini
cp lib/LCS/BV.pm /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/lib/LCS/BV.pm
cp t/10_basic_lcs.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/t/10_basic_lcs.t
cp t/10_basic_llcs.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/t/10_basic_llcs.t
cp t/10_countbits.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/t/10_countbits.t
cp t/author-minimum-version.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/t/author-minimum-version.t
cp t/author-pod-coverage.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/t/author-pod-coverage.t
cp xt/10_profile.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/xt/10_profile.t
cp xt/50_bits_bench.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/xt/50_bits_bench.t
cp xt/50_diff_bench.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/xt/50_diff_bench.t
cp xt/50_diff_time.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/xt/50_diff_time.t
cp xt/50_llcs_time.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/xt/50_llcs_time.t
cp xt/author-pod-spelling.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/xt/author-pod-spelling.t
cp xt/author-pure-ascii.t /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/xt/author-pure-ascii.t
cp xt/bench.txt /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/xt/bench.txt
Building /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5
Detecting project name from directory name.
Retrieving meta data from lib/LCS/BV.pm.
Name: LCS::BV
Abstract: Bit Vector (BV) implementation of the Longest Common Subsequence (LCS) Algorithm
Version: 0.09
Writing MANIFEST file
Writing release tests: xt/minilla/minimum_version.t
Writing release tests: xt/minilla/cpan_meta.t
Writing release tests: xt/minilla/pod.t
Writing release tests: xt/minilla/spelling.t
Writing release tests: xt/minilla/permissions.t
[zwNXqdc5] $ /Users/helmut/perl5/perlbrew/perls/perl-5.20.1/bin/perl -I/usr/local/bin -I/usr/local/lib -I/Users/helmut/perl5/perlbrew/perls/perl-5.20.1/lib/site_perl/5.20.1/darwin-2level -I/Users/helmut/perl5/perlbrew/perls/perl-5.20.1/lib/site_perl/5.20.1 -I/Users/helmut/perl5/perlbrew/perls/perl-5.20.1/lib/5.20.1/darwin-2level -I/Users/helmut/perl5/perlbrew/perls/perl-5.20.1/lib/5.20.1 -I. Build.PL
Creating new 'Build' script for 'LCS-BV' version '0.09'
[zwNXqdc5] $ /Users/helmut/perl5/perlbrew/perls/perl-5.20.1/bin/perl -I/usr/local/bin -I/usr/local/lib -I/Users/helmut/perl5/perlbrew/perls/perl-5.20.1/lib/site_perl/5.20.1/darwin-2level -I/Users/helmut/perl5/perlbrew/perls/perl-5.20.1/lib/site_perl/5.20.1 -I/Users/helmut/perl5/perlbrew/perls/perl-5.20.1/lib/5.20.1/darwin-2level -I/Users/helmut/perl5/perlbrew/perls/perl-5.20.1/lib/5.20.1 -I. Build build
cp lib/LCS/BV.pm blib/lib/LCS/BV.pm
t/10_basic_lcs.t .............. ok
t/10_basic_llcs.t ............. ok
t/10_countbits.t .............. ok
t/author-minimum-version.t .... ok
t/author-pod-coverage.t ....... ok
xt/10_profile.t ............... No subtests run
xt/50_bits_bench.t ............ ok
xt/50_diff_bench.t ............ No subtests run
xt/50_diff_time.t ............. No subtests run
xt/50_llcs_time.t ............. Can't use string ("Chrerrplzon") as an ARRAY ref while "strict refs" in use at /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5/blib/lib/LCS/BV.pm line 37.
xt/50_llcs_time.t ............. Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run
xt/author-pod-spelling.t ...... ok
xt/author-pure-ascii.t ........ ok
xt/minilla/cpan_meta.t ........ skipped: Test::CPAN::Meta required for testing META.yml
xt/minilla/minimum_version.t .. skipped: Test::MinimumVersion::Fast required for testing perl minimum version
xt/minilla/permissions.t ...... skipped: Test::PAUSE::Permissions is not installed.
xt/minilla/pod.t .............. ok
xt/minilla/spelling.t ......... skipped: Test::Spellunker is not installed.
Test Summary Report
-------------------
xt/10_profile.t (Wstat: 0 Tests: 0 Failed: 0)
Parse errors: No plan found in TAP output
xt/50_diff_bench.t (Wstat: 0 Tests: 0 Failed: 0)
Parse errors: No plan found in TAP output
xt/50_diff_time.t (Wstat: 0 Tests: 0 Failed: 0)
Parse errors: No plan found in TAP output
xt/50_llcs_time.t (Wstat: 65280 Tests: 0 Failed: 0)
Non-zero exit status: 255
Parse errors: No plan found in TAP output
Files=17, Tests=1655, 287 wallclock secs ( 0.17 usr 0.08 sys + 283.88 cusr 2.51 csys = 286.64 CPU)
Result: FAIL
Removing /Users/helmut/github/perl/LCS-BV/.build/zwNXqdc5
real 4m47.772s
user 4m44.936s
sys 0m2.835shelmut@Helmuts-MacBook-Pro:~/github/perl/LCS-BV$ time milla test --release
[DZ] building distribution under .build/pb0pzsMzPW for installation
[@Milla/NameFromDirectory] guessing your distribution name is LCS-BV
[DZ] beginning to build LCS-BV
[DZ] guessing dist's main_module is lib/LCS/BV.pm
[@Milla/LicenseFromModule] guessing from lib/LCS/BV.pm, License is Software::License::Perl_5
[@Milla/LicenseFromModule] Copyright -8 Helmut Wollmersdorfer <helmut.wollmersdorfer@gmail.com>
[@Milla/ExtraTests] rewriting author test xt/author/pod-syntax.t
[@Milla/Prereqs::FromCPANfile] Parsing 'cpanfile' to extract prereqs
[@Milla/StaticInstall] setting x_static_install to 1
[DZ] writing LCS-BV in .build/pb0pzsMzPW
[@Milla/CopyFilesFromBuild] Copied .build/pb0pzsMzPW/META.json to META.json
[@Milla/CopyFilesFromBuild] Copied .build/pb0pzsMzPW/LICENSE to LICENSE
[@Milla/CopyFilesFromBuild] Copied .build/pb0pzsMzPW/Build.PL to Build.PL
[@Milla/ReadmeAnyFromPod] overriding README.md in root
Creating new 'Build' script for 'LCS-BV' version '0.09'
cp lib/LCS/BV.pm blib/lib/LCS/BV.pm
t/10_basic_lcs.t ............ ok
t/10_basic_llcs.t ........... ok
t/10_countbits.t ............ ok
t/author-minimum-version.t .. ok
t/author-pod-coverage.t ..... ok
t/author-pod-syntax.t ....... ok
All tests successful.
Files=6, Tests=1634, 4 wallclock secs ( 0.14 usr 0.02 sys + 3.52 cusr 0.05 csys = 3.73 CPU)
Result: PASS
[DZ] all's well; removing .build/pb0pzsMzPW
real 0m5.539s
user 0m5.358s
sys 0m0.278s
@karenetheridge Thanks for the link. I knew about it (participated Berlin 2015), but it's hard to find.
From Lancaster Consensus:
We also (briefly) discussed how some tools like Dist::Zilla are using AUTHOR_TESTING distinct from RELEASE_TESTING.
and
AUTHOR_TESTING: if true, tests are being run as part of an author's personal development process; such tests may or may not be run prior to release. CPAN clients must not set this variable. Distribution packagers (ppm, deb, rpm, etc.) should not set this variable.
For me "author's personal development process" implies running at authors will. With such convenience tools like Milla and Minilla the author delegates the control more or less to the tool. That's fine, if the tools work as expected, which they do in most of the cases. The Lancaster Consensus is not clear about the exact handling of author tests, nor does it provide rules for including or excluding the subdirectories t and xt, or file extensions *.t.
Roughly Milla executes t/*.t, Minilla t/*.t and xt/*.t. For me the workaround will be either renaming the files to *.pl or protect them with skip and new environment variables like DEVEL_TESTING, BENCHMARK, PROFILING and even WOLLMERS as a last resort.
A better solution would be, that the related tools Zilla, Milla, Minilla and others agree to the same default behaviour and semantics of testing options.
I would not expect that tests under
xt/would be executed during build or whatever per default.Migrated from Dist::Milla and tried to release: