databrickslabs / dataframe-rules-engine

Extensible Rules Engine for custom Dataframe / Dataset validation
Other
134 stars 30 forks source link

Adding unit tests and test reporting to the dataframe-rules-engine #6

Closed stikkireddy closed 4 years ago

stikkireddy commented 4 years ago

added jacoco plugin into sbt added ruletype enum to make it easier to manage rule types for testing added entry to contributing.md on how to submit pull requests added information to readme on how to run tests altered gitignore to include plugins.sbt which is for jacoco test reports altered build.sbt to include jacoco test reporting & added scalatest added 6 tests regarding validation.

stikkireddy commented 4 years ago

@GeekSheikh let me know what you think 😄

GeekSheikh commented 4 years ago

Will review ASAP. Thank you!

GeekSheikh commented 4 years ago

Hey, Sri,

This is great, thanks for building up the scaffolding and throwing in a few tests. If you get some time, could you review this PR? It fixed a bug with the output report, if you get time, please add some additional tests to validate the output df.

Also, is the coverage sufficient for all the tests without explicitly doing the test with a group by and without a group by? I saw the validations on the group by cols, but not 100% sure that covers all cases.

We also need to add an agg column in each test to validate the agg expressions vs simple column expressions. We also need to add a test for a complex column to each test case such as col("retail_price" - col("scan_price") to ensure we're validating expressions properly.

I believe these would be simple, quick additions, do you think you will have time to add them soon? Appreciate the contribution.

Summary