search

awslabs / python-deequ

Python API for Deequ
Apache License 2.0
676 stars 131 forks source link

isContainedIn not working for String having single quote (') in it #79

Closed sbbagal13 closed 5 months ago

sbbagal13 commented 2 years ago

Describe the bug Column value check with isContainedIn DQ check not working for string having single quote (') example : Can't Proceed

To Reproduce Run the isContainedIn constraints on any column which contains value with single quote e.g. .isContainedIn("REASON", ["Already Has "," Can't Proceed"])

Expected behavior constraints result Success but its failing and code getting generated like below

ComplianceConstraint(Compliance(REASON contained in Already Has ,Can't Proceed, CLOSE_LOST_REASON IS NULL OR CLOSE_LOST_REASON IN ('Already Has','Can''t Proceed'),None))

sbbagal13 commented 2 years ago

I found below code in Scala code https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/checks/Check.scala

val valueList = allowedValues .map { _.replaceAll("'", "''") } .mkString("'", "','", "'")

why single quotes are replacing by two single quotes ?

eycho-am commented 5 months ago

Closing as this issue was fixed in https://github.com/awslabs/deequ/pull/484 for issue https://github.com/awslabs/deequ/issues/462. Added unit tests in a recent PR https://github.com/awslabs/deequ/pull/532 https://github.com/awslabs/deequ/blob/master/src/test/scala/com/amazon/deequ/checks/CheckTest.scala#L538-L544