Nike-Inc / spark-expectations

A Python Library to support running data quality rules while the spark job is running⚡
https://engineering.nike.com/spark-expectations
Apache License 2.0
161 stars 37 forks source link

Integrating the notifications to Zoom Webhook #83

Closed amaldevk88 closed 3 months ago

amaldevk88 commented 6 months ago

Description

I've expanded the notification capability within the spark_expectations library to include Zoom alongside the existing Slack and Team notifications. This improvement allows users to receive updates on data quality job execution status across Zoom,Slack and Teams platforms.

Related Issue

Please link to the issue here: #82

Motivation and Context

The motivation behind this change is to broaden the reach of data quality framework execution status notifications. While Slack is a popular communication platform, some organizations prefer or exclusively use Zoom. By incorporating Zoom notifications into the spark_expectation library, we enhance its usability for a wider audience, ensuring that data quality job execution status alerts are accessible across multiple collaboration tools.

How Has This Been Tested?

To ensure the reliability and functionality of the Zoom notification feature, I thoroughly tested the following aspects: Unit Tests:- Implemented unit tests to verify the correctness of the Zoom notification implementation. Integration Testing: Conducted integration tests by simulating different data quality scenarios and validating the generation and delivery of notifications to Teams. Environment: Tested the feature in various environments like databricks, local IDEs as well as to ensure compatibility and functionality across different setups. Message Formatting: Checked the appearance and readability of messages sent to Zoom to ensure they display correctly. This comprehensive testing approach aimed to validate the feature's effectiveness, compatibility, and correctness before integrating it into the spark_expectations library.

Screenshots (if appropriate):

Link to Unit test document:- Test.Results.Proofs.-.SE.Zoom.Notif.Feature.zip

Types of changes

Checklist:

asingamaneni commented 5 months ago

@amaldevk88 Looks like the lint checks failed, can you please correct and run make cov in your local to validate everything is right. Thanks!

amaldevk88 commented 5 months ago

@amaldevk88 Looks like the lint checks failed, can you please correct and run make cov in your local to validate everything is right. Thanks!

lint check issues are fixed.

asingamaneni commented 5 months ago

@amaldevk88 can you please fix the unittests.

amaldevk88 commented 5 months ago

regex pattern mismatch issues in Unit test are fixed. Kindly rerun unit test.

amaldevk88 commented 5 months ago

just rebased code and pushed again. Kindly rerun

amaldevk88 commented 3 months ago

Please review and approve the merge