Language-agnostic tests

[SignpostMarv]

Would I be alone in thinking that having a unified, language-agnostic test suite would be beneficial?

I've recently been thinking over the issue of unit tests for the different implementations (mostly just the PHP & JS ones as you might guess). A quick google suggested that language-agnostic test suites are few & far between. So I rolled my own :P

It struck me that one would need to define a series of tests for VerbalExpressions that can be read in as many languages as possible. Parsing XML isn't exactly straight-forward in PHP or JS, so I went with JSON.

So with the test suites being defined in JSON, it would seem like a good idea to ensure that the test suites were well-defined (so each implementation would know what to expect of the structure of the files). For this we have JSON Schema and a grunt task I cooked up last night (this issue would've been posted last night, but unicorns got in the way). This allows test definers to be confident that when a new version goes out there should only be issues with implementations when the test schema changes (i.e. if grunt jsonschema fails, don't push it).

The basic schema I've implemented in the proof-of-concept examples allows for:

• multiple tests per file (no restrictions on file names beyond any silliness that would prevent an implementation from parsing it)
• abstract call stacks & arguments (only a single call stack for all languages)
• default expected output + language-specific outputs (if for whatever reason an implementation needs to generate different regex strings to be most performant whilst doing the same job)

Thoughts/comments/suggestions?

• example test suite definition.
• test suite schema
• PHP Implementation of language agnostic tests

[jehna]

For JS we can use VerEx[method]() and for PHP VerEx->$method() (or similar). But are you aware of any restrictions on other implementations? E.g. how would c# implementation handle the method calling (as, with my small experience with c# I can't recall how to use variable name functions)?

[SignpostMarv]

I shall have a tinker over the weekend.

[SignpostMarv]

Dynamic invocation can be performed in c#- here's the proof-of-concept.

It's only proof-of-concept because DynamicInvocationTestCases() features a hard-coded version of the getRegex.json example- I had tinkered with adding a nuget package, but couldn't figure out the syntax to get the files to be copied to the build directory so one can parse them with something like Newtonsoft's JSON parser, hence skipping to the proof-of-concept.

[lukewestby]

Dynamic invocation isn't possible in Elm, and I'm sure this is true of some of the other strongly-typed languages as well. I'm wondering, would a standard JSON format for verbal expressions operations be a good step forward in creating a cross-language test suite? This could be used in conjunction with additional JSON data for expected results to create a repeatable test suite that each language can deal with just in the way it deals with parsing JSON in general.

[SignpostMarv]

Re: elm, we could perhaps compile JSON definitions to elm tests ?

[lukewestby]

This is definitely also possible, I know some others in the community have experimented with building tools for bridging JSON and Elm type definitions and decoders. I'll reach out to some folks and report back with what I find out!

[SignpostMarv]

Also having a browse through the docs, could the test suite be run with qunit & Elm worker ?

[SignpostMarv]

Have just implemented a proof-of-concept for JSVerbalExpressions that parses the tests JSON & generates an abstract syntax tree to run the tests; I'm wondering how many of the other implementation languages have libraries that support creating scripts from ASTs ?

[SignpostMarv]

I've just implemented a working AST-based test generator for PHPVerbalExpressions- see output in files suffixed with "DynamicallyGeneratedTest.php"

[SignpostMarv]

Further experiments in the matter of generating tests; swap out a use statement in PHP to get a json-serializable representation of the call stack on an instance of VerbalExpressions: TraceverbalExpressions