Custom constraints on dataframes

[FRosner]

Problem

If you want to add a constraint that is not implemented, it would be nice to open a new issue at DDQ! Until the new constraint is implemented and released, you should be able to implement the constraint as a custom one by yourself.

Draft

There should be a custom constraint function, which takes the constraint name and a function operating on the dataframe and returning either a success or a failure message, as a parameter.

• If the function returns a success message, we will mark the constraint as successful
• If the function returns a failure message, we will mark the constraint as a failure
• If the function throws an exception, we will mark the constraint as errored

Check(customers)
  .hasNumRows(_ >= 3)
  .hasUniqueKey("id")
  .custom(
    "number of columns = 100",
    (df: DataFrame) => if (df.columns.size == 100) Right("number of columns match") else Left(s"number of columns were ${df.columns.size}")
  ).run()

Documentation

https://github.com/FRosner/drunken-data-quality/wiki/Drunken-Data-Quality-4.1.0#custom-constraints