Add data quality part for parking_sensor_synapse using great expectations library

[cchenshu]

Type of PR

• Code changes

Purpose

• Add the data validataion part for parking_sensor_synapse using great expectations library, following the similar logic in the databricks.
• For both 02_standardize and 03_transform notebook, add the following steps for data validation.
  0. Create mount point path for spark job
  1. Configure DataContext
  2. Create a BatchRequest based on dataframe
  3. Define Expecation Suite and corresponding Data Expectations
  4. Configure a checkpoint and run Expectation suite using checkpoint

     Does this introduce a breaking change? If yes, details on what can break

     No

     Author pre-publish checklist

• [x] Executed test to prove my fix is effective or new feature works
• [ ] No PII in logs
• [ ] Made corresponding changes to the documentation

Validation steps

• Run the notebooks: 02_standardize, 03_transform

Issues Closed or Referenced

• Closes #issue_number
• References #issue_number

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB