search

lordofthejars / nosql-unit

NoSQL Unit is a JUnit extension that helps you write NoSQL unit tests.
Other
383 stars 123 forks source link

New Comparison Strategy using JSONassert #159

Open connollyst opened 8 years ago

connollyst commented 8 years ago

I've written a new MongoComparisonStrategy based on the JSONassert project and really find it helps me debug my failing tests a lot quicker.

Background

For background, the default comparison strategy builds a query using the expected data as an example. So, if everything matches then great, but if any of the expected JSON doesn't match the actual object, the query fails. The only feedback you get is that the expected JSON wasn't found. This makes it very difficult to debug what the actual JSON was and what needs to be corrected.

How it Works

This new JsonComparisonStrategy queries each document by only it's _id field, and then compares the expected JSON to the actual JSON. The feedback is much more useful, for example:

com.lordofthejars.nosqlunit.core.NoSqlAssertionError: _id : xyz/two, authorization.admins
Expected: -rw
     got: --w

This tells you that the record with _id xyz/two for the subdocument field authorization.admins was expected to be -rw but was --w.

For comparison (har har) the default comparison strategy would report:

com.lordofthejars.nosqlunit.core.NoSqlAssertionError: Object # { "_id" : "xyz/two" , "project" : "xyz" , "name" : "two" , "displayName" : "Group Two" , "description" : "Group Two, of Namespace XYZ." , "order" : 2 , "retired" : true , "hidden" : true , "authorization" : { "users" : "-r-" , "admins" : "-rw"}} # is not found into collection [entity_groups]

My Questions

Two questions.

lordofthejars commented 8 years ago

Absolutely. Thank you very much for your contribution, go ahead with the required changes.

Thanks.

connollyst commented 8 years ago

Awesome, I'll be in touch with a PR.

connollyst commented 8 years ago

Hey Alex,

I've got my implementation ready and am trying to write some unit tests. I was basing them off of MongoFlexibleComparisonStrategyTest to keep the code/test style consistent, but those tests are failing on my machine. Do they pass for you?

The error is 

shouldThrowAnExceptionIfDifferentValuesInFlexibleStrategy(com.lordofthejars.nosqlunit.mongodb.MongoFlexibleComparisonStrategyTest)  Time elapsed: 0.04 sec  <<< FAILURE!
java.lang.AssertionError: Expected exception: com.lordofthejars.nosqlunit.core.NoSqlAssertionError
    at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:32)
    at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.evaluate(AbstractNoSqlTestRule.java:72)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:158)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)

The exception is declared as expected like so: @Test(expected=NoSqlAssertionError.class) If I take out the Exception expectation part, then the test fails because the exception is thrown.

I tried removing that kind of Exception expectation and tried using the @Rule ExpectedException.. approach, no luck.

It almost seems like the @ShouldMatchDataSet is executed in some different scope than the @Test(expected=NoSqlAssertionError.class) Exception check. Just guessing though.

Any thoughts? If those tests work for you, might it be something about your environment setup?

Cheers, Sean

lordofthejars commented 8 years ago

Currently I am on vacation. I will try next week.

Thanks

El 20 ago. 2016 6:58 p. m., "Sean Connolly" notifications@github.com escribió:

Hey Alex,

I've got my implementation ready and am trying to write some unit tests. I was basing them off of MongoFlexibleComparisonStrategyTest to keep the code/test style consistent, but those tests are failing on my machine. Do they pass for you?

The error is

shouldThrowAnExceptionIfDifferentValuesInFlexibleStrategy(com.lordofthejars.nosqlunit.mongodb.MongoFlexibleComparisonStrategyTest) Time elapsed: 0.04 sec <<< FAILURE! java.lang.AssertionError: Expected exception: com.lordofthejars.nosqlunit.core.NoSqlAssertionError at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:32) at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.evaluate(AbstractNoSqlTestRule.java:72) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:158) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)

The exception is declared as expected like so: @Test(expected= NoSqlAssertionError.class) If I take out the Exception expectation part, then the test fails because the exception is thrown.

I tried removing that kind of Exception expectation and tried using the @Rule ExpectedException.. approach, no luck.

It almost seems like the @ShouldMatchDataSet is executed in some different scope than the @Test(expected=NoSqlAssertionError.class) Exception check. Just guessing though.

Any thoughts? If those tests work for you, might it be something about your environment setup?

Cheers, Sean

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lordofthejars/nosql-unit/issues/159#issuecomment-241211056, or mute the thread https://github.com/notifications/unsubscribe-auth/ABcmYXD1yPh4tYyQrn7VsAuGF4eVvtNyks5qhzIdgaJpZM4IyfmH .

connollyst commented 7 years ago

Hi Alex,

Any update on this? I'd love to contribute this pull request if possible.

Cheers, Sean

lordofthejars commented 7 years ago

You can contribute for sure, now I can release a new version pretty fast.

connollyst commented 7 years ago

Great, good to hear.

Can you comment on the unit tests? When I check out the master branch and run the unit tests, many fail with the error I mentioned above. Want to make sure I'm starting with everything working correctly before I start mucking around.

Cheers, Sean

lordofthejars commented 7 years ago

Test fixed, you can start working on this.

connollyst commented 7 years ago

Hi Alex,

I still see a number of other failing tests. Are these failing on your side too?

-Sean

lordofthejars commented 7 years ago

Can you list them? I execute some of them and they worked.

El dv., 10 de febr. 2017 a les 12:54, Sean Connolly (< notifications@github.com>) va escriure:

Hi Alex,

I still see a number of other failing tests. Are these failing on your side too?

-Sean

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/lordofthejars/nosql-unit/issues/159#issuecomment-278925671, or mute the thread https://github.com/notifications/unsubscribe-auth/ABcmYRiHVjNn8-24AV3wc13lk2Zjn1VIks5rbE_sgaJpZM4IyfmH .

connollyst commented 7 years ago

Sure, 42 of the 62 tests in nosqlunit-mongodb run, 5 fail. WhenMultipleMongoObjectsAreAnnotatedWithInject seems to run indefinitely and so the remaining tests never complete. The 5 failing tests are below.

WhenExpectedDataShouldBeCompared.less_expected_collection_than_database_collection_should_fail:

Expected :Expected collection names are [col1] but insert collection names are [col1, col3]
Actual   :Expected collection names are [col1] but insert collection names are [col3, col1]

WhenExpectedDataShouldBeCompared.expected_collection_item_has_less_attributes_than_database_collection_item_attribute_should_fail:

Expected :Expected DbObject and insert DbObject have different keys: Expected: [name] Inserted: [name, surname]
Actual   :Expected DbObject and insert DbObject have different keys: Expected: [name] Inserted: [surname, name]

WhenManagedMongoDbRuleIsRegistered.mongo_server_should_throw_an_exception_if_mongo_location_is_not_found:

Exception in thread "Thread-6" java.lang.IllegalStateException: Mongodb [/example--dbpathmongo-dbpath--port27017--logpathlogpath] could not be started. Next console message was thrown: Cannot run program "/example/bin/mongod" (in directory "target/mongo-temp"): error=2, No such file or directory
    at com.lordofthejars.nosqlunit.mongodb.ManagedMongoDbLifecycleManager$ProcessRunnable.prepareException(ManagedMongoDbLifecycleManager.java:354)
    at com.lordofthejars.nosqlunit.mongodb.ManagedMongoDbLifecycleManager$ProcessRunnable.run(ManagedMongoDbLifecycleManager.java:311)
    at java.lang.Thread.run(Thread.java:745)

WhenMongoDbRuleIsRegistered.should_clean_previous_data_and_insert_new_dataset_with_clean_insert_strategy:

com.lordofthejars.nosqlunit.core.NoSqlAssertionError: Expected collection names are [collection1, collection2] but insert collection names are [collection1, collection2, col1]

    at com.lordofthejars.nosqlunit.core.FailureHandler.createFailure(FailureHandler.java:7)
    at com.lordofthejars.nosqlunit.mongodb.MongoDbAssertion.checkCollectionsName(MongoDbAssertion.java:51)
    at com.lordofthejars.nosqlunit.mongodb.MongoDbAssertion.strictAssertEquals(MongoDbAssertion.java:32)
    at com.lordofthejars.nosqlunit.mongodb.DefaultComparisonStrategy.compare(DefaultComparisonStrategy.java:17)
    at com.lordofthejars.nosqlunit.mongodb.DefaultComparisonStrategy.compare(DefaultComparisonStrategy.java:10)
    at com.lordofthejars.nosqlunit.core.AbstractCustomizableDatabaseOperation.executeComparison(AbstractCustomizableDatabaseOperation.java:15)
    at com.lordofthejars.nosqlunit.mongodb.MongoOperation.compareData(MongoOperation.java:103)
    at com.lordofthejars.nosqlunit.mongodb.MongoOperation.databaseIs(MongoOperation.java:96)
    at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.assertExpectation(AbstractNoSqlTestRule.java:255)
    at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.evaluate(AbstractNoSqlTestRule.java:79)
    at com.lordofthejars.nosqlunit.mongodb.integration.WhenMongoDbRuleIsRegistered.should_clean_previous_data_and_insert_new_dataset_with_clean_insert_strategy(WhenMongoDbRuleIsRegistered.java:163)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.junit.runners.Suite.runChild(Suite.java:127)
    at org.junit.runners.Suite.runChild(Suite.java:26)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

WhenMongoDbRuleIsRegistered.should_assert_if_expected_data_is_strict_equal:

com.lordofthejars.nosqlunit.core.NoSqlAssertionError: Expected collection names are [collection1, collection2] but insert collection names are [collection1, collection2, col1]

    at com.lordofthejars.nosqlunit.core.FailureHandler.createFailure(FailureHandler.java:7)
    at com.lordofthejars.nosqlunit.mongodb.MongoDbAssertion.checkCollectionsName(MongoDbAssertion.java:51)
    at com.lordofthejars.nosqlunit.mongodb.MongoDbAssertion.strictAssertEquals(MongoDbAssertion.java:32)
    at com.lordofthejars.nosqlunit.mongodb.DefaultComparisonStrategy.compare(DefaultComparisonStrategy.java:17)
    at com.lordofthejars.nosqlunit.mongodb.DefaultComparisonStrategy.compare(DefaultComparisonStrategy.java:10)
    at com.lordofthejars.nosqlunit.core.AbstractCustomizableDatabaseOperation.executeComparison(AbstractCustomizableDatabaseOperation.java:15)
    at com.lordofthejars.nosqlunit.mongodb.MongoOperation.compareData(MongoOperation.java:103)
    at com.lordofthejars.nosqlunit.mongodb.MongoOperation.databaseIs(MongoOperation.java:96)
    at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.assertExpectation(AbstractNoSqlTestRule.java:255)
    at com.lordofthejars.nosqlunit.core.AbstractNoSqlTestRule$1.evaluate(AbstractNoSqlTestRule.java:79)
    at com.lordofthejars.nosqlunit.mongodb.integration.WhenMongoDbRuleIsRegistered.should_assert_if_expected_data_is_strict_equal(WhenMongoDbRuleIsRegistered.java:82)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.junit.runners.Suite.runChild(Suite.java:127)
    at org.junit.runners.Suite.runChild(Suite.java:26)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
    at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

I want to get all tests running & passing before I make any code changes so I know I don't break anything by adding this new feature.

Cheers, Sean