Tau.feat 2031.incorporate gol testsuite

tabulon commented 4 years ago

This PR brings in the tooling for incorporating and running GOL's own test-suite directly within GLM's repo along the lines as described in #26, except:

we've opted for author-only at this time; so the tests (as well as the corersponding Makefile) will appear under xt/
some minor differences in namings

Note that this PR does NOT include any real tests, but only the tooling that enables generating some automatically from GOL's own test-suite.

Here's how the repo will look like after applyingv this PR:

perl-Getopt-Long-More/
  ...
  maint/
    adapt-gol-test           # adapt (to GLM) a single test script  from GoL's test-suite 
    adapt-gol-tests         # adapt (to GLM) a bunch of test scripts from GoL 
    fetch-gol-tests          # fetch GOL's test suite from CPAN. 
  xt/
     gol-authentic/            # automatically built by 'make'   # - (not included by this PR)
     gol-conformance/      # automatically built by 'make'   # - (not included by this PR)
     Makefile

Running make within the xt/directory will generate two subdirectories there:

gol-authentique/ : containing the unmodified GoL test-suite, automatically fetched from CPAN;
gol-conformance/ : containing automatically patched versions of the same GOL tests, modified to employ GLM, instead of GOL.

Although they are automatically built, those sub-directoies and their contents (*.t) will also checked in to the GLM git repo (later). Imho, this has several advantages that outweigh the usual culprits.

Also, by keeping the original GOL test suite around (under gol-authentique/), we are able to catch the case when something is broken for both GLM and GOL, in which case the blame might well be on GOL itself.

The mini-build process is handled by a separate Makefile under the test directory xt/, in a way currently independent of the dzil build. Here's how:

$ (cd xt/ && make)
$ git add xt/gol-*

And then the usual test flows work as expected, such as:

# prove works ...
$ prove t/ xt/gol-*

# dzil is also able to pick up the tests
$ dzil xtest

BTW:

So as not to overcharge xt/Makefile, the actual work is done by a three simple scripts under maint/.
The only external dependency (other than make itself) of this new mini test-build is cpanm.

perlancar commented 4 years ago

Is xt/gol-conformance/gol-compat.t passing on your side? Do we even need this test?

tabulon commented 4 years ago

Just to make sure: You have it failing for both gol-authentique and gol-conformance, right?

On my machine, they pass (but see below).

However, the gol-compat.t did give me some trouble in the beginning (for both authentique and confromance), to the point that I asked myself the same question: Do we even need this thing?

Re-installing GoL via cpanm solved the issue for me:

cpanm Getopt::Long

Before that, the test was having trouble when doing a require "newgetopt.pl".

So I also looked into what the heck it was to see if we really need it.

It turns out it's a compatibility test for GoL's own historical baggage. GoL itself appears to be a rewrite (on Perl5) of a thingy that was named "newgetopt.pl" (on Perl 4).

And gol-compat.t is a test to see if newgetopt.pl (which is now a stub which just calls GoL) still works (under Perl5). And it does :-)

So => do we need it? Not really... (unless we do end up forking GoL.. which we might).

But since it did pass in the end (once having installed GoL from CPAN), I just kept it; and then forgot to mention the ordeal. Sorry..

I now realize that the reason it was initially failing to find newgetopt.pl was probably because that thingy (which is part of the GoL distro) is not installed with PERL CORE (GoL is a dual-life package)...

However, if we do end up moving these gol-* tests back under t/, we should either:

a) omit (or skip) that test b) Keep a copy of newgetopt.pl around (for conformance, have it call GLM, instead of GoL) and make sure gol-compat.t can actually find it (add a copule lines with FindBin or something)

BTW, just checked manually... newgetopt.pl does work when it's modified to use Getopt::Long::More.

So, I vote for (b). It's not really hard to do since we have the tooling in place. Besides, it's a good air-pump for hubris :-)

If you agree, I will go ahead and code this into the tooling .

What do you think?

On 9 Apr 2020, at 4:32, perlancar wrote:

Is xt/gol-conformance/gol-compat.t passing on your side? Do we even need this test?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/perlancar/perl-Getopt-Long-More/pull/29#issuecomment-611293640

perlancar commented 4 years ago

Yes I had it failing on both gol-authentique & gol-conformance. It seems like the core version of GoL does not include newgetopt.pl, which is installed if we install the CPAN version of GoL. Now it passes for both.

Since this is an author test, the author can just reinstall GoL from CPAN. I'm fine with keeping the test.

perlancar / perl-Getopt-Long-More

Tau.feat 2031.incorporate gol testsuite #29