awslabs / python-deequ

Python API for Deequ
Apache License 2.0
713 stars 134 forks source link

Unable to resolve column names with a '.' in the name #99

Open Tom-Hudson opened 2 years ago

Tom-Hudson commented 2 years ago

Describe the bug When attempting to use any check on a field that has a '.' in the name such as:

.isComplete("column.name")

You will get an exception like:

Exception: Verification checks failed: [{'check_status': 'Error', 'check_level': 'Error', 'constraint_status': 'Failure', 'check': 'Verification checks', 'constraint_message': "org.apache.spark.sql.AnalysisException: cannot resolve '`column.name`' given input columns: [column.name];

To Reproduce Steps to reproduce the behavior:

  1. Create a data source with a column name that has a '.' in such as 'column.name'
  2. Create a verification suite that includes the check .isComplete("column.name")
  3. Run the code
  4. See the error

Expected behavior The field name should be resolved, allowing checks to be performed