search

Nike-Inc / spark-expectations

A Python Library to support running data quality rules while the spark job is running⚡
https://engineering.nike.com/spark-expectations
Apache License 2.0
148 stars 32 forks source link

[BUG] Meta row dq_results section is empty in error table #92

Open blaine12100 opened 1 month ago

blaine12100 commented 1 month ago

Describe the bug When data is written to error table, there is no information on which rule(s) caused this row to be written to the error table

To Reproduce Steps to reproduce the behavior:

  1. Follow steps given in the github example to create rules and enable them
  2. Load the necessary data and enable writing to the error table
  3. When the process is complete, check the meta_row_dq_results section.

Expected behavior

I expect the row to have a list of rule(s) which caused it to be put into this table.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context

The user configuration object has been created by following the examples in the documentation. Is the data not written due to it not being implemented or the toggling of a specific user configuration option.