awslabs / python-deequ

Python API for Deequ
Apache License 2.0
713 stars 134 forks source link

Add severity optional parameter to constraints #180

Open gracemiguel opened 9 months ago

gracemiguel commented 9 months ago

Describe the solution you'd like In my use of pydeequ, I would like to be able to express the severity of a constraint(low, medium, high). The closest thing that exists to this currently is the CheckLevel, but that does not meet my requirements. I would like the user to be able to input severity as an optional parameter to a constraint, not the check object, so that this can be reflected in the VerificationResult dataframe.

Describe alternatives you've considered I have tried adding severity from a higher level perspective, from a dataframe perspective, but that isn't granular enough. In my use, I am adding constraints on on check object and creating a VerificationResult dataframe from that. If this feature could be added to the constraints I would be really grateful. I am also open to suggestions.

chenliu0831 commented 9 months ago

It might be good to share a few examples (stubbed interface) for your use-case, which should be useful for us to consider whether it should be added in Scala land

gracemiguel commented 9 months ago

I think something like this in the constraint would help identify the severity. By having it as a parameter for the constraint it can be directly assigned to the constraint and the dataframe it is being performed on.

image

In the verification dataframes(successMetricsAsDataFrame and checkResultsAsDataFrame) I would want that severity parameter to be a column like this:

image