great-expectations / great_expectations

Always know what to expect from your data.
https://docs.greatexpectations.io/
Apache License 2.0
9.86k stars 1.52k forks source link

[BUG] GX unable to send notification to Teams channel after data validation in the checkpoint #10363

Closed Erua-chijioke closed 1 week ago

Erua-chijioke commented 2 weeks ago

Describe the bug Using Great expectations version 1.0.2, I created an action inside a checkpoint to send a notification to a teams channel through a webhook but It is unable to do that and displays this error:

To Reproduce

**create data source, asset, batch definition and expectation suite**
dataframe = spark.sql("SELECT * FROM xxx")
context = gx.get_context(project_root_dir="/dbfs/xxx")

data_source_name = "my_data_source"
data_asset_name = "my_dataframe_data_asset"
batch_definition_name = "my_batch_definition"
data_source = context.data_sources.add_or_update_spark(name=data_source_name)
data_asset=data_source.add_dataframe_asset(name=data_asset_name)
batch_definition = data_asset.add_batch_definition_whole_dataframe(batch_definition_name)

suite_name = "my_expectation_suite"
suite = gx.ExpectationSuite(name=suite_name)
suite = context.suites.add(suite)

table_row_count=dataframe.count()
table_row_count_expectationn = gxe.ExpectTableRowCountToEqual(value=table_row_count)
suite.add_expectation(table_row_count_expectationn)

**Retrieve the expectation suite and create validation definition**
expectation_suite_name = "my_expectation_suite"
expectation_suite = context.suites.get(name=expectation_suite_name)

definition_name = "my_validation_definition"
validation_definition = gx.ValidationDefinition(
        data=batch_definition, suite=expectation_suite, name=definition_name)
validation_definition = context.validation_definitions.add(validation_definition)

 validation_definitions = [
    context.validation_definitions.get("my_validation_definition")]

validation_notification_microsoft_teams_webhook = "xxx"

**create an action list**
action_list = [
    # This Action sends a Team Notification if an Expectation fails.
    gx.checkpoint.MicrosoftTeamsNotificationAction(
        name="send_microsoft_teams_notification_on_validation_result",
        teams_webhook= validation_notification_microsoft_teams_webhook,
        notify_on= "all",
    ),
    # This Action updates the Data Docs static website with the Validation
    #   Results after the Checkpoint is run.
    gx.checkpoint.UpdateDataDocsAction(
        name="update_all_data_docs",
    ),
]

**create a checkpoint**
checkpoint_name = "my_checkpoint"
    checkpoint = gx.Checkpoint(
        name=checkpoint_name,
        validation_definitions=validation_definitions,
        actions=action_list,
        result_format={"result_format": "COMPLETE"},
    )
    context.checkpoints.add(checkpoint)

batch_parameters = {"dataframe": dataframe}

validation_results = checkpoint.run(
    batch_parameters=batch_parameters,)

**Below is the error message I get**
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7048a166-73a8-4fdf-aa6c-fb34a0b4805e/lib/python3.10/site-packages/great_expectations/render/renderer/microsoft_teams_renderer.py:114, in MicrosoftTeamsRenderer._render_data_docs_links(self, data_docs_pages)
    112 if docs_link_key == "class":
    113     continue
--> 114 docs_link = data_docs_pages[docs_link_key]
    115 report_element = self._get_report_element(docs_link)
    116 elements.append(report_element)

TypeError: list indices must be integers or slices, not ValidationResultIdentifier

Expected behavior The checkpoint is expected to send a notification to a teams channel through the teams webhook provided.

Environment:

adeola-ak commented 2 weeks ago

Hi @Erua-chijioke, thank you for highlighting this issue. I will share it with the team and follow up

adeola-ak commented 1 week ago

the fix has been merged and will be in our next release, thanks again for bringing this to our attention as well as the descriptive repro steps!

Erua-chijioke commented 1 week ago

@adeola-ak You are welcome! Thank you very much for your timely response and the fix! I will look forward to it in the next release.