I am trying to implement a solution using the pydeequ's constraint based method .hasDataType but it can only check a column for a particular data type only and causes a Failure
Lets say a column has been defined to contain Integer type data, for some fields the value for that column may or may not be present, then according to this constraint .hasDataType('column', ConstrainableDataTypes.Integer) the result will be failure alongwith a value between 0-1 signifying the percentage of Integral values in the column.
Queries -
Will it be possible to implement multiple data type checks for a particular column?
If not, then how can we handle the above scenario?
In case of a column with integers and null values, can checking the column against NumericType help?
Scenario -
.hasDataTypebut it can only check a column for a particular data type only and causes aFailureIntegertype data, for some fields the value for that column may or may not be present, then according to this constraint.hasDataType('column', ConstrainableDataTypes.Integer)the result will be failure alongwith a value between 0-1 signifying the percentage of Integral values in the column.Queries -
NumericTypehelp?