Error trying to send notifications to Teams workflow/webhook using Prefect Cloud automations

[bjorhn]

Bug summary

We've just migrated from the old Teams webhooks to the new Teams workflows for handling notifications. I've created a block which works when I use it in a Prefect flow:

teams_webhook_block = MicrosoftTeamsWebhook.load("teams-analytics")
teams_webhook_block.notify("Hello from Prefect!")

However, when I try to use this block in an automation in Prefect Cloud, I get the following error when looking at the event feed: "notification_log": "[ERROR]: There are no service(s) to notify"

I'm not sure how to debug this further.

We're running the latest version of Prefect (3.0.2) and in order to perform the migration from the old webhook blocks to the new workflow block types for Teams, I deleted the old block, re-registered the notification block types (using prefect block register --module prefect.blocks.notifications) and then configured the new one.

Version info (prefect version output)

Version:             3.0.2
API version:         0.8.4
Python version:      3.11.9
Git commit:          c846de02
Built:               Fri, Sep 13, 2024 10:48 AM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         cloud
Pydantic version:    2.9.1
Integrations:
  prefect-dbt:       0.6.1
  prefect-snowflake: 0.28.0
  prefect-shell:     0.3.0
  prefect-azure:     0.4.0

Additional context

Full event feed error message:

{
  "id": "83ad1077-5437-4115-92f2-08a0014e6782",
  "account": "*removed*",
  "event": "prefect-cloud.automation.action.failed",
  "occurred": "2024-09-17T12:27:19.951Z",
  "payload": {
    "action_index": 0,
    "action_type": "send-notification",
    "invocation": "574a13f8-ffcc-4939-9e77-a03e67fcb2b2",
    "reason": "Notification failed",
    "notification_log": "[ERROR]: There are no service(s) to notify"
  },
  "received": "2024-09-17T12:27:19.951Z",
  "related": [
    {
      "prefect.resource.id": "prefect.block-document.3b7e2d2d-f35d-4cd7-a403-cd495ea2aae6",
      "prefect.resource.role": "block",
      "prefect.resource.name": "teams-analytics"
    },
    {
      "prefect.resource.id": "prefect.block-type.ms-teams-webhook",
      "prefect.resource.role": "block-type"
    }
  ],
  "resource": {
    "prefect.resource.id": "prefect-cloud.automation.de11bb80-53bb-4121-aa9c-db1a60a8b554",
    "prefect.resource.name": "Test",
    "prefect-cloud.trigger-type": "event",
    "prefect-cloud.posture": "Reactive"
  },
  "workspace": "*removed*"
}

[jallison-relaytx]

I am having the same issue with identical notification_log messages.

[codingtacos]

I am having similar issues. The URL changed in Microsoft Teams and when we went to update it or try to create a brand new block in the Cloud UI we have the same errors.

Version: 2.16.9 API version: 0.8.4 Python version: 3.11.9 Git commit: 083def52 Built: Thu, Apr 4, 2024 3:11 PM OS/Arch: darwin/arm64 Server type: cloud

15:06:27.171 | DEBUG | prefect.client - Connecting to API at https://api.prefect.cloud/api/accounts/xxxxxx 15:06:28.718 | ERROR | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 2 pending items.

[MartinGJespersen]

We are having the same issue and get the same error as @bjorhn in the event feed.