Nike-Inc / spark-expectations

A Python Library to support running data quality rules while the spark job is running⚡
https://engineering.nike.com/spark-expectations
Apache License 2.0
161 stars 37 forks source link

Add teams notification feature #58

Closed BharatKumarEY closed 10 months ago

BharatKumarEY commented 10 months ago

Integrating the notifications to Teams Channel

Description

I've expanded the notification capability within the spark_expectations library to include Microsoft Teams alongside the existing Slack notifications. This improvement allows users to receive updates on data quality job execution status across both Slack and Teams platforms.

Related Issue

Please link to the issue here: https://github.com/Nike-Inc/spark-expectations/issues/56

Motivation and Context

The motivation behind this change is to broaden the reach of data quality framework execution status notifications. While Slack is a popular communication platform, some organizations prefer or exclusively use Microsoft Teams. By incorporating Teams notifications into the spark_expectation library, we enhance its usability for a wider audience, ensuring that data quality job execution status alerts are accessible across multiple collaboration tools.

How Has This Been Tested?

To ensure the reliability and functionality of the Teams notification feature, I thoroughly tested the following aspects: Unit Tests:- Implemented unit tests to verify the correctness of the Teams notification implementation. Integration Testing: Conducted integration tests by simulating different data quality scenarios and validating the generation and delivery of notifications to Teams. Environment: Tested the feature in various environments like databricks, local IDEs as well as to ensure compatibility and functionality across different setups. Message Formatting: Checked the appearance and readability of messages sent to Teams to ensure they display correctly. Link to Unit test doccument:- This comprehensive testing approach aimed to validate the feature's effectiveness, compatibility, and correctness before integrating it into the spark_expectations library.

Screenshots (if appropriate):

Test Results & Proofs - SE Teams Notif Feature.zip Please refer the attachments section

Types of changes

Checklist:

codecov[bot] commented 10 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (6d546bb) 100.00% compared to head (171113c) 100.00%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #58 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 22 22 Lines 1441 1441 ========================================= Hits 1441 1441 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.