Improvements to testcase format and tooling

[svaarala]

The testcase format with included files and expect string etc is quite old and could use a few improvements. The two main directions are:

• Extending the print-and-expect based approach.
• Adopting an assert-based approach.

Both have merits, see discussion in https://github.com/svaarala/duktape/pull/1073.

For print-and-expect:

• Remove include files and allow tests to assume a Test object with a bunch of helpers always exists. Also provide a Test module for e.g. Node.js where possible, so that comparison is easy.
• Inline expect syntax, e.g. print(-0); //> -0 so that expect strings and tests can mostly be close to each other.
• More compact syntax for metadata, e.g. //! custom: true.
• Better handling of known issues, e.g. known issue files which specify allowed diffs (or exact outputs for known issues).
• Helper for try catch wrapping, e.g. Test.run(function () { ... }) with automatic stack trace printing etc.
• Automatic uncaught error printing for tests with just global level code, which would eliminate the need for a try-catch boilerplate in many tests.
• Use console.log() rather than print() so that cross checking with other engines (V8 especially) is easier. Or maybe Test.print()?
• Add a metadata key for a "barebones" test which won't get the Test binding; this allows low level lexer tests and such to continue to work with minimal engine assumptions.
• Add a metada key strict: true to indicate a test should run as a strict program. One can also use "use strict"; but that has some issues with prepending a test framework.
• Command line tools for prepping test cases and running them with an arbitrary engine. This functionality is now tied deep into runtests.js which is awkward. The Test module can be pre-minimized to a single line function so that it can be prepended more easily without needing a minifier.

[svaarala]

With regards to tooling, runtests.js is a pretty awkward tool overall. Current plan is to split its functionality into two:

The main primitive would be a Python2 utility to prepare and optionally run a testcase:

• Testcase file is given as an argument.
• The test framework (Test binding or similar) is automatically located relative to the utility location. The test framework would be pre-minified to a single-line file so that it can be prepended to the testcase without affecting line numbering (which matters to some tests). Pre-minification avoids a minified dependency which is simple but annoying.
• A prepared testcase file is output either to requested output file, or temporary file if running the test directly. The prepared testcase includes the test framework and the testcase and should be directly runnable with duk, nodejs, etc.
• Optionally, the testcase emitted is executed using duk, nodejs, or other engine. Differences to expected output are reported (hopefully with a better syntax than now).
• Known issues would be automatically scanned, and could be implemented as a set of whitelisted outputs so that if they match exactly, the known issue can be noted in the test output. (Current handling is just to note that it's OK that the test fails, but the difference to expected is ignored.)

Runtests.js can then be kept and will be a bit smaller. It might still run API tests which involve compilation and linking. Or the unit test tool could also support API tests and be capable of compiling and linking individual tests.

[svaarala]

1094 adds util/runtest.py which prepares and runs a testcase, with:

• Support for YAML test metadata
• Support for inline expect syntax //>hello world
• Support for known issue files describing exact "known bad" outputs rather than blanket metadata indicating "this test can fail"
• Support for use_strict: true metadata flag
• Support for intended_uncaught: true metadata flag
• Ensures print() and console.log() are available for all engines
• Adds a placeholder global Test binding which can be later extended with more test framework code

[fatcerberus]

What is the purpose of the use_strict flag? Wouldn't putting a real strict mode directive at the top of the file work just as well?

[svaarala]

The testcase prologue is prepended into the prepared test so a real strict mode directive isn't effective.

Ideally the test preparation code would parse the real strict mode directive and place the prologue after it but that's more work than just a metadata flag.

[fatcerberus]

Ah, didn't think of that, that makes sense :)