search

stencila / encoda

↔️ A format converter for Stencila documents
https://stencila.github.io/encoda/
Apache License 2.0
35 stars 9 forks source link

Use generative testing #57

Open nokome opened 5 years ago

nokome commented 5 years ago

One of the current testing approaches is to to have many fixtures and automatically iterate through them, doing conversions and comparisons between formats as implied by file name extensions. That's OK but it's cumbersome maintaining an adhoc set of fixtures. Particulary when there are changes to encodings.

We have recently added a custom jest matcher which tests for invert-ability: https://github.com/stencila/convert/blob/90586a9820598ebb36c45169377c24777a001b89/tests/matchers.ts#L9-L30

Currently we are just passing that some sample node trees. However, give we have a schema, we could use json-schema-faker, to generate many node trees and check that the compiler can invert them (ie. unparse and then parse them and get the same node back) e.g. 

await expect(jats).toInvert(sampleNode)
alex-ketch commented 5 years ago

Came across this relevant article last night, so thought I'd share.

nokome commented 4 years ago

Just did a little more research on this. It still feels to me that json-schema-faker is the best approach as it's one less step removed from the schema itself: https://github.com/json-schema-faker/json-schema-faker/blob/master/docs/USAGE.md.

But the randomness of this is probably not what we need right now for Thema visual regression testing e.g. it would take some work to ensure only one of each node type. I think a more manual approach to developing examples with "one of everything" would be better.