PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.79k stars 1.55k forks source link

Add SNS Notify Block #13046

Open EmilRex opened 1 year ago

EmilRex commented 1 year ago

Proposal

It would be great to be able to use AWS's Simple Notification Service (SNS) to deliver Prefect notifications. Specifically, I'd like to be able to publish a message to a SNS topic as a notify capability. In order to be ultimately useful, this block would need to be usable in a Cloud automation.

Example

from prefect_aws.sns import SNS

def create_block():
    block = SNS(
        topic_arn="arn:aws:sns:us-east-1:123456789123:PrefectNotifications"
    )
    block.save("prefect-notifications")

def send_message(message: str):
    block = SNS.load("prefect-notifications")
    block.notify("Send help!")
jbnitorum commented 1 year ago

This block should be able to utilize a AWS credential block to generate the apprise path rather than having the user store keys in the block.

Also it would be really helpful if we could pass context of the flow triggering the notification to the SNS topic in the form of a json payload. Since SNS has many programmatic applications beyond just typical textual notifications this seems like a good place to offer this option. It also helps bridge parity to Cloud 1.0 where custom webhooks could send a json payload with flow_run_id, state, etc.

I think it could be possible to create stringified json from the current jinja templating, but its certainly not as clean as the 1.0 implementation.

image