Test Suite: Ensure GLM passes GoL test suite

tabulon commented 4 years ago

GLM aims to be a drop-in replacement for GoL.

One way to ensure we don't break GoL compatibility is to somehow run GoL's own test suite on GLM.

Don't exactly know how to go about this. Ideas welcome.

Here are the alternative ways I can think of to achieve this :

a) Manual (one off) incorporation
- copy the GoL test suite (*.t) in two directories under t/ => e.g.: t/gol t/gol-compat
- In each test script that matches t/gol-compat/*.t => search & replace (GoL => GLM)
b) Semi-automatic incorporation Same steps as the manual method above, but try to automate the process so that we can easily update the GoL tests with new releases of GoL. I can think of two ways of doing this:
- b-1) as part of the make / build
- b-2) as part of running the test suite
c) Via a specialized Test module (to be created or found, if one exists), something like which does the same thing in an automated (and generic) fashion.

I will probably go ahead and propose a quick and dirty manual solution based on (a). We can see about the others later, unless there are better ideas.

BTW, this exercise will probably prompt us to modify GLM to be a subclass of GoL; or at least, more specifically, to pond a GLM::Parser which is a subclass of GoL::Parser (the OO interface of GoL).

tabulon commented 4 years ago

I know have a working solution for semi-automatically incorporating and running the GoL test-suite on GLM.

The good news is => with a few additional changes, GLM is indeed able to pass the GoL test-suite! Yeyy :-)

(BTW, as expected, this did require subclassing GoL::Parser and a few other changes that help expose the same public API as GoL. )

The mediocre news is that the proposed solution is a bit amateurish and does not easily fit the dzil workflow (due to my ignorance)...

perl-Getopt-Long-More/
  lib/
    ...
  maint/
    adapt-gol-test
    adapt-gol-tests
    fetch-gol-tests
  t/
     gol-authentic/
     gol-conformance/
     01-basic.t
     Makefile

There are two new auto-generated subdirectories under t/ that result from a mini build process as part of the authoring workflow :

gol-authentic/ : contains the unmodified GoL test-suite, automatically fetched from CPAN;
gol-conformance/ : contains automatically patched versions of the same GOL tests, modified to employ GLM, instead of GOL.

Although they are automatically built, those sub-directoies and their contents (*.t) are also checked in to the GLM git repo. Imho, this has several advantages that outweigh the usual culprits.

Also, by keeping the original GOL test suite around (under gol-authentic/), we are able to catch the case when something is broken for both GLM and GOL, in which case the blame might well be on GOL itself.

The mini-build process is handled by a separate Makefile under the test directory t/, in a way currently independent of the dzil build. Here's how:

$ (cd t/ && make)
$ git add t/gol-*

And then the usual test flows work as expected, such as:

# prove works ...
$ prove t/ t/gol-*

# dzil is also able to pick up the tests
$ dzil test

BTW:

So as not to overcharge t/Makefile, the actual work is done by a three simple scripts under maint/.
The only external dependency (other than make itself) of this new mini test-build is cpanm.

perlancar commented 4 years ago

Nice work.

Although automation is good, I'd probably vote for manual (a) or semi-automatic. I'd be okay with the test being author/release test only too.

tabulon commented 4 years ago

OK, great. In the end, it did end up being semi-automatic anyways, with the Makefile approach described above.

Also, before I submit any PR :

If these tests will be author only, should we place them in xt/ then?
Do you see any merit in testing multiple versions of GOL? (for later)

perlancar / perl-Getopt-Long-More

Test Suite: Ensure GLM passes GoL test suite #26