homogenization of testsuite syntax

[lefessan]

The syntax of the testsuite is not homogeneous, so it makes it difficult to use other tools than autoconf to work with it.

The goal of this PR is to make it more homogeneous, and test/dev some other tooling to simplify running the testsuite and fixing issues (while still using autoconf testsuite for mainstream use).

[GitMensch]

Because of other pending commits I've checked in all of the testsuite changes that were related and done, please update the mirror and branch, enabling us to see what's "missing"; do you have any thoughts on the open points outlined above?

[lefessan]

I will update this PR on top of your commits, but here is a few points:

• I have a small prototype of tool that is able to run a large part of the tests, it's currently very basic, but I hope it will soon have some nice "modern" features. Given my agenda, I am only working on it from time to time, but it is on my top list when I have time for GnuCOBOL, as I am struggling more and more with the testsuite;

• About the current syntax: if we want tools to be able to use the test files, we should use as much M4 macros as possible, instead of using shell commands (that are extremely hard to parse...).

  • So, it is always better to use AT_DATA instead of echo (my tool for example can only handle single-line echo )
  • It would probably be better NOT to use [ ... ] to scope macros inside run-if-fail or run-if-pass, because [...] is usually used to store unparsed string, and '[' & ']' are escape characters within them.
  • I don't know yet what to do with ifs, but there are not so many of them (update: use AT_CHECK for these ones)

[GitMensch]

Most of the conditional "if failing, then ..." are about 32/64 bit, you may come up with an GC_TEST32 GC_TEST64 macro (ideally defined in testsuite.at), the ABI that does not match would then be ignored, the other one executed. This should reduce the issues and create not so many conflicts (writing and using more macros needs to be postponed until everything is merged to trunk).

[lefessan]

Actually, I would prefer not to introduce new macros, as the tool is based on the knowledge of a small fixed set of macros.

I actually got rid of these tests using AT_CHECK and the run-if parts for then/else.

[GitMensch]

Hm, I'll have a look at your suggestion next week after the update and push.

[lefessan]

I still need also to review it, in particular the replacements of if by AT_CHECK. I haven't been careful enough on them.

[lefessan]

For information, the program I wrote is here: https://github.com/ocamlpro/autofonce It's main features:

• running the testsuite in parallel with minimal output
• creating a test by capturing a command execution

It requires this PR to be merge to fully work on gnucobol (once the workflows are fixed)

[codecov-commenter]

Codecov Report

Merging #82 (b2f7da1) into gcos4gnucobol-3.x (7fd5d49) will not change coverage. The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@                Coverage Diff                 @@
##           gcos4gnucobol-3.x      #82   +/-   ##
==================================================
  Coverage              65.27%   65.27%           
==================================================
  Files                     31       31           
  Lines                  56542    56542           
  Branches               14767    14767           
==================================================
  Hits                   36908    36908           
  Misses                 13828    13828           
  Partials                5806     5806           

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[lefessan]

I have a new version of the tool that is much more tolerant to the syntax, I am going to submit an updated version of this PR with much fewer changes.

[GitMensch]

feel free to check in

[lefessan]

Merged !