Incorporating and merging with the compliance test suite

cburgmer / json-path-comparison

Comparison of the different implementations of JSONPath and language agnostic test suite.

https://cburgmer.github.io/json-path-comparison/

GNU General Public License v3.0

84 stars 26 forks source link

Incorporating and merging with the compliance test suite #130

Open gregsdennis opened 1 year ago

gregsdennis commented 1 year ago

Both this project and the CTS have a lot of useful test cases. I think it would be ideal to maintain these in a single place.

To that end, I'd like to propose merging the CTS test cases into this project. This project seems to be ideal because it already contains test harnesses for many well-known implementations.

To do this, I'd like to add some properties to the test cases to indicate what the spec says:

spec-valid - A boolean that determines whether the path is valid (can be parsed) according to the spec. There are a number of test cases in the CTS to ensure that the implementation returns an error when it should.
spec-result - The expected result as defined by the spec. I think it's still valuable to collect consensus results as well.

There may be more needed, but this is what I can think of so far.

gregsdennis commented 1 year ago

Related to #127

glyn commented 1 year ago

@gregsdennis If the CTS test cases were merged into this project, would the CTS project still be maintained separately? I don't think every implementor will want to add their implementation to this project (e.g. if it is closed source) just to find out if it passes the CTS.

gregsdennis commented 1 year ago

would the CTS project still be maintained separately?

No, that project could be retired.

I don't think every implementor will want to add their implementation to this project (e.g. if it is closed source) just to find out if it passes the CTS.

There's no requirement to add an implementation to the comparison to see if it passes the CTS. The "CTS" portion of the project is really just a list of test cases and expected results, just like the current CTS. It could still be run privately.

Benefits of doing this are (not limited to):

we maintain the test cases in one place
there are a lot of spec-invalid test cases in this project that are useful
we get to see how closely (these) implementations align with the spec
those implemnetations which have been implemented here get a free CTS implementation and report