Consider Support for the TP3 Test Suite

[spmallette]

The testing pattern for Ogre for TP2 seems to have been to re-implement the tests from gremlin-groovy. While this may ultimately be the pattern to follow for Ogre in TP3, JVM languages have the ability to take advantage of the gremlin-test suite which provides access to hundreds of tests to clearly validate the "gremlin flavor" against its specification. In this way, there is a consistent way to test Gremlin from a traversal perspective. From Ogre's perspective, the benefit to using this suite is that it gets a complete battery of tests without having the test code to maintain. Implementations simply provide a Traversal object to the gremlin-test suite in order to have its results validated.

It is worth keeping in mind that these tests will only cover the Gremlin "Process" features (i.e. traversals). Ogre will need to maintain tests for other aspects of the language (e.g. transactions, graph strategies, etc.)

In addition to gremlin-java and gremlin-groovy, gremlin-scala (a project external to TinkerPop) currently uses the test suite to its advantage.

It would be nice if there was a way for Ogre to use this test suite. It does mean that Ogre would need to somehow leverage JUnit in its testing (as gremlin-test is based on that framework). There's some other Java-interop issues as well related to Java annotations and other odds and ends that will need to be sorted to get this going.

In the end, perhaps this may not be possible for some technical reason....if so, Ogre will need to maintain its own set of tests.

[fitzoh]

I started experimenting with this in a branch @ https://github.com/Fitzoh/ogre/tree/testSuite

I added the lein-junit plugin, and a test/java folder which currently contains a single class which has copy-pasted traversals from the tp3 repo. Instead starting to dig in and track down the errors, I thought I should check if there's anything immediately obvious as to why it's not working.

fitz@fitz-ThinkPad-T420s:~/Workspace/ogre$ lein junit Testsuite: org.clojurewerkz.ogre.ThingTest Tests run: 3, Failures: 0, Errors: 3, Time elapsed: 0.015 sec Testcase: g_V_both_hasXlabel_softwareX_dedup_byXlangX_name(org.clojurewerkz.ogre.ThingTest): Caused an ERROR null java.lang.NullPointerException at com.tinkerpop.gremlin.AbstractGremlinTest.setup(AbstractGremlinTest.java:57) Testcase: g_V_both_dedup_name(org.clojurewerkz.ogre.ThingTest): Caused an ERROR null java.lang.NullPointerException at com.tinkerpop.gremlin.AbstractGremlinTest.setup(AbstractGremlinTest.java:57) Testcase: g_V_both_name_orderXa_bX_dedup(org.clojurewerkz.ogre.ThingTest): Caused an ERROR null java.lang.NullPointerException at com.tinkerpop.gremlin.AbstractGremlinTest.setup(AbstractGremlinTest.java:57) Test org.clojurewerkz.ogre.ThingTest FAILED JUnit tests failed.

[spmallette]

You might be on to something here. I'm not sure how I didn't come across the lein-junit plugin when I was trying to figure out how to to do this (or maybe I did and don't remember where I got blocked).

I'm not sure exactly why you are getting this error, but I'm thinking it has something to do with your approach to how you are trying to utilize the test suite. Basically, you can't run the test on its own as you have it. You have to run that test as part of a custom Gremlin test suite, which auto-injects graph instances. In this way, a Gremlin language implementation can test their implementation against lots of different Gremlin structure implementations.

So, you extend a test as you did, supply the Gremlin-flavor representation of the requested Traversal and the suite handles the rest. In other words, your ThingTest should be returning the Ogre representation of that Traversal. Then, you need a test suite class that extends from AbstractGremlinSuite - probably called OgreProcessStandardSuite. In that you need supply the set of tests (in this case all you have is ThingTest, but presumably you would have implementations for all of the gremlin test suite) to the constructor of AbstractGremlinSuite.

Once you have the suite defined, you need to create a test case to use the suite. For that you need to choose a Graph implementation you want to run the suite with. I'd just worry about TinkerGraph for now. It would basically look like this:

@RunWith(OgreProcessStandardSuite.class)
@OgreProcessStandardSuite.GraphProviderClass(provider = TinkerGraphGraphProvider.class, graph = TinkerGraph.class)
public class OgreTinkerGraphProcessStandardTest {
}

Run that test with junit and it should execute all the tests you added to the OgreProcessStandardSuite.

Thanks for digging into this. Please let me know if there are other questions.

[fitzoh]

Just pushed some updates to my branch.

First issue I ran into was TinkerGraphGraphProvider not being available (it's in src/test), so I just copy-pasted that for the time being.

I'm now running into an issue with TinkerGraph not opting in to Ogre tests:

$ lein junit
Testsuite: org.clojurewerkz.ogre.OgreTinkerGraphProcessStandardTest
Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.008 sec
------------- Standard Error -----------------
Review the testsToExecute given to the test suite as the following are missing: [class org.clojurewerkz.ogre.ThingTest]
------------- ---------------- ---------------
Testcase: initializationError(org.clojurewerkz.ogre.OgreTinkerGraphProcessStandardTest):    Caused an ERROR
The suite will not run for this Graph until it is publicly acknowledged with the @OptIn annotation on the Graph instance itself
java.lang.Exception: The suite will not run for this Graph until it is publicly acknowledged with the @OptIn annotation on the Graph instance itself
    at com.tinkerpop.gremlin.AbstractGremlinSuite.validateOptInToSuite(AbstractGremlinSuite.java:164)
    at com.tinkerpop.gremlin.AbstractGremlinSuite.<init>(AbstractGremlinSuite.java:100)
    at com.tinkerpop.gremlin.AbstractGremlinSuite.<init>(AbstractGremlinSuite.java:86)
    at org.clojurewerkz.ogre.OgreProcessStandardSuite.<init>(OgreProcessStandardSuite.java:57)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
Test org.clojurewerkz.ogre.OgreTinkerGraphProcessStandardTest FAILED
Testsuite: org.clojurewerkz.ogre.ThingTest
Tests run: 3, Failures: 0, Errors: 3, Time elapsed: 0.016 sec
Testcase: g_V_both_hasXlabel_softwareX_dedup_byXlangX_name(org.clojurewerkz.ogre.ThingTest$StandardTest):   Caused an ERROR
null
java.lang.NullPointerException
    at com.tinkerpop.gremlin.AbstractGremlinTest.setup(AbstractGremlinTest.java:57)
Testcase: g_V_both_dedup_name(org.clojurewerkz.ogre.ThingTest$StandardTest):    Caused an ERROR
null
java.lang.NullPointerException
    at com.tinkerpop.gremlin.AbstractGremlinTest.setup(AbstractGremlinTest.java:57)
Testcase: g_V_both_name_orderXa_bX_dedup(org.clojurewerkz.ogre.ThingTest$StandardTest): Caused an ERROR
null
java.lang.NullPointerException
    at com.tinkerpop.gremlin.AbstractGremlinTest.setup(AbstractGremlinTest.java:57)
Test org.clojurewerkz.ogre.ThingTest FAILED
JUnit tests failed.

[spmallette]

Sorry this is kinda painful, but I think this is a good exercise - once we get this working I'm going to turn this issue into TP3 documentation. :)

Two ways to overcome this issue:

1. drop this constructor: https://github.com/Fitzoh/ogre/blob/756eabb2d1f3e405cb9c5c32c8f302e52e6d7d86/test/java/org/clojurewerkz/ogre/OgreProcessStandardSuite.java#L64 and make sure that the last argument gremlinFlavorSuite is set to true. That should bypass that initialization error.
2. Ignore my previous advice to extend AbstractGremlinSuite and extend ProcessStandardSuite...again, make sure you pass true for that argument. You can see what I think is a good example of how to extend from ProcessStandardSuite by looking at the groovy example: https://github.com/tinkerpop/tinkerpop3/blob/be31bbdc208cb94bffd756bb45287a7d3a496107/gremlin-groovy-test/src/main/java/com/tinkerpop/gremlin/process/GroovyProcessStandardSuite.java

Hopefully, that will finally get the tests to run.

[fitzoh]

And we have running tests. Not expecting it to be accepted yet, but i'm going to open a PR to make this easier to follow/comment on.

$ lein junit
Testsuite: org.clojurewerkz.ogre.OgreTinkerGraphProcessStandardTest
Tests run: 3, Failures: 1, Errors: 0, Time elapsed: 0.253 sec
------------- Standard Output ---------------
Testing: [GraphStep(vertex), VertexStep(BOTH,vertex), HasStep([~label,eq,software]), DedupStep(value(lang)), PropertiesStep([name],value)]
         [TinkerGraphStep(vertex), VertexStep(BOTH,vertex), HasStep([~label,eq,software]), DedupStep(value(lang)), PropertiesStep([name],value)]
Testing: [GraphStep(vertex), VertexStep(BOTH,vertex), DedupStep, PropertiesStep([name!],value)]
         [TinkerGraphStep(vertex), VertexStep(BOTH,vertex), DedupStep, PropertiesStep([name!],value)]
Testing: [GraphStep(vertex), VertexStep(BOTH,vertex), PropertiesStep([name],property), OrderStep(lambda), DedupStep, PropertyValueStep]
         [TinkerGraphStep(vertex), VertexStep(BOTH,vertex), PropertiesStep([name],property), OrderStep(lambda), DedupStep, PropertyValueStep]
------------- ---------------- ---------------
------------- Standard Error -----------------
Review the testsToExecute given to the test suite as the following are missing: [class org.clojurewerkz.ogre.ThingTest]
------------- ---------------- ---------------
Testcase: g_V_both_dedup_name(org.clojurewerkz.ogre.ThingTest$StandardTest):    FAILED
expected:<6> but was:<0>
junit.framework.AssertionFailedError: expected:<6> but was:<0>
    at com.tinkerpop.gremlin.process.graph.step.filter.DedupTest.g_V_both_dedup_name(DedupTest.java:36)
    at com.tinkerpop.gremlin.AbstractGremlinSuite.runChild(AbstractGremlinSuite.java:236)
    at com.tinkerpop.gremlin.AbstractGremlinSuite.runChild(AbstractGremlinSuite.java:39)
Test org.clojurewerkz.ogre.OgreTinkerGraphProcessStandardTest FAILED
Testsuite: org.clojurewerkz.ogre.ThingTest
Tests run: 0, Failures: 0, Errors: 0, Time elapsed: 0.005 sec
JUnit tests failed.

[spmallette]

nice - this is a good sign.

[spmallette]

@Fitzoh came with the nice model for allowing this to work. It is implemented in full on the master branch right now against TinkerPop 3.2.0-incubating.