[SPIKE] 1 day to form an opinion on what we need rulesets to do, and thus which technology to use

[robredpath]

We know that we want to test data against sets of rules. We know that there are at least 3 ways that we can do this:

• Schematron
• @andylolz 's YAML format
• @Bjwebb 's format, that's the official IATI ruleset format

We should take 1 day to work out:

• What we are going to want to test with rulesets
• What implementing each of them would look like (eg can the DI module play a part?)
• What the downsides and limitations of each one is

[robredpath]

I'll do the 'what do we want to test' part, @kindly will evaluate the options

[andylolz]

@andylolz 's YAML format

NB these days I’m using gherkin for this. Here are some sample tests.

[robredpath]

Thanks @andylolz ! How're you finding it?

[andylolz]

Yeah, gherkin is fine for this I think (I’ve previously used it for actual BDD on ODI projects.) I expect tests run slower than they would with a bespoke thing like the IATI rulesets tester… But gherkin/cucumber are pretty widely used / well-documented / maintained etc, so that’s the trade-off I suppose.

[kindly]

I have been doing some research on the suggested options. I am excluding @andylolz yaml format as he is now using gherkin anyway and it has no overriding benefit over the other options.

Schematron

This is a standard way of doing these kind of rulesets in xml and is commonly used in conjunction with xmlschema. It is different from xmlschema as you do not have to specify the whole structure of the document but just certain constraints or rules that you want it to conform to.

I found this pdf a good primer to it.

Pros:

• An actual standard of how to write rules using xpath as expected.
• In XML and the standard is XML
• Rule language easier to write than I expected.
• Has good support in libraries across languages i.e libxml supports it and that has binding is many languages.
• Know about xml structure to save writing long xpaths each time as you can just start a particular position in the document.
• Only need to know schematron to write rules, no need to write any code.

Cons

• Not very human readable rulesets.
• In XML!
• Error output is also in XML so not trivial to parse out the errors you want to see.
• Some non clarity about what subset of schematron you can use with particular libraries.
• Good documentaton is hard to come by and not sure all the functions you can use in the rule language.
• Not actual code so is a chance there is rule you want that can't be expressed in its rule language.
• There may be a lot of work converting the current rulesets to this format.

Iati Ruleset Language (made by Ben)

Pros:

• In the standard itself.
• In JSON.
• Fast

Cons:

• Not very easy to read.
• Non standard structure and way of expressing rules.
• Need to implement again in each programming language.

Gherkin

Pros:

• Very human readable rulesets.
• Neat way of expressing which conditions to actually check the rule under.
• Structure of a Gherkin is standard and very many parsers to read it.
• Very flexible and easy ways to write code in order to make different kind of rules.

Cons:

• Actual code of rulesets have to be written in each programming language.
• There may be a lot of repetition of some of the rules.
• New types of rules will need to be written by people who have a good understanding of code. There is not any xml specific ones (apart from by @anyholz) that we could easily reuse.
• Easy to make typo when specifying rules as have to use exactly the same human readable expression.
• There may be a lot of work converting the current rulesets to this format.
• May run slower due to human readable structure.

Conclusion

I have come round to the idea of using gherkin, especially as the human readability of the rulesets is so much better then any other option and at least the structure of the rules are standard. Some part of me thinks schematron is the "proper" way to do this (as its a standard and has lots of support) but it will take some learning for any coder to understand how to write them. It is also a taste thing as I do not have much experience of XML and gherkins look much easier to get started with especially as @andylolz has already begun this work.

[andylolz]

I wonder if it’s worth mentioning:

Obviously this is all for the IATI-specific flavour of CoVE, buuut… I suppose it’s possible you might at some point want something similar to this for 360giving / OpenContracting? Is that a consideration? (It’s possible this was baked into the “In XML!” con against schematron.)

[robredpath]

@andylolz yup, we're having a conversation about how this cuts across standards at the moment; this is a theme that's either come up or is on the horizon for both OCDS and 360, so we're hoping to find a solution that can be widely applied.