[FEATURE] Filter condition is ignored when filtering records based on row-level checks

[eapframework]

I tried the filtering data based on row-level checks and it is working fine. New column is added with true or false based on the row-level checks. But if rules has where filter condition, filter condition is ignored.

It is marking all the records as false ignoring the filter applied to the rule. Is it possible to extract bad records after applying where filter in row-level checks.

[eycho-am]

Hi @eapframework, thank you for raising this issue. Could you provide an example of the behavior you're seeing and the behavior you expect/want instead?

We're actually looking into a similar issue - here's an example unit test using the Completeness analyzer:

For the dataframe:

  def getDfCompleteAndInCompleteColumns(sparkSession: SparkSession): DataFrame = {
    import sparkSession.implicits._

    Seq(
      ("1", "a", "f"),
      ("2", "b", "d"),
      ("3", "a", null),
      ("4", "a", "f"),
      ("5", "b", null),
      ("6", "a", "f")
    ).toDF("item", "att1", "att2")
  }

This test checks the row-level results

    "return row-level results for columns filtered" in withSparkSession { session =>

      val data = getDfCompleteAndInCompleteColumns(session)

      val completenessAtt2 = Completeness("att2", Option("att1 = \"a\""))
      val state = completenessAtt2.computeStateFrom(data)
      val metric: DoubleMetric with FullColumn = completenessAtt2.computeMetricFrom(state)

      data.withColumn("new", metric.fullColumn.get).collect().map(_.getAs[Boolean]("new")) shouldBe
        Seq(true, false, false, true, false, true)
    }

Using the verification suite on a similar test:

+----+----+----+-----+
|item|att1|att2|rule1|
+----+----+----+-----+
|   1|   a|   f| true|
|   2|   b|   d|false|
|   3|   a|null|false|
|   4|   a|   f| true|
|   5|   b|null|false|
|   6|   a|   f| true|
+----+----+----+-----+

Here we can see that the rows that EITHER are filtered out (rows 2,5 att1 is not a) or fail the check (row 3 is null) are marked as false. Would you expect rows 2,5 to show true/None in this case?

[eapframework]

Thanks for your response. This is the same issue I am facing.
I am expecting rows 2,5 to show true because those are not failed records

Expected result:

     +----+----+----+-----+
     |item|att1|att2|rule1|
     +----+----+----+-----+
     |   1|   a|   f| true|
     |   2|   b|   d| true|
     |   3|   a|null|false|
     |   4|   a|   f| true|
     |   5|   b|null| true|
     |   6|   a|   f| true|
     +----+----+----+-----+

[eycho-am]

Thanks for your feedback @eapframework,

We're working through different use cases for different users and I'm planning a PR for this soon. We're planning on providing a configuration so users can set filtered rules as Null or True - so setting this configuration to True should meet your use-case. I'll tag you on the PR once we have that out as well.

[eycho-am]

Hi @eapframework, we've merged PR #532 addressing this issue for Uniqueness and Completeness analyzers and another one open for other analyzers: #535

Please let us know if you have any feedback on these PRs and add comments or open a PR if this doesn't quite meet your use-case.

[eapframework]

Hi @eycho-am , Thanks for merging the PR to address the use-case. Sorry for the delayed response. It is working fine except when values are null in the column.

For example:

1. Rule: containsCreditCardNumber("creditno", == 1.0).where("indicator == 'b'") Values:

   credit_no             indicator
   4012888888881881          null     

   For above values, status is marked as false but expected true

2. hasPattern("account_no", "[0-9]{7}".r).where("indicator == 'b'") Values:

   account_no               indicator
   1288888                    null    

   For above values, status is marked as false but expected true

3. hasCompleteness("numhist", => 0.8).where("indicator == 'b'") Values:

   num_hist              indicator
   18                          null 

   For above values, status is marked as blank but expected true