Investigate options for dbt failure notifications

jkarpen commented 3 weeks ago

Goal for this issue is to explore different options for having dbt cloud failure notifications go to a space where multiple team members (including client agencies ideally) can triage and assign the fix.

[ ] Slack integration (failure creates a Slack message visible to the whole DSE team)
[x] Teams notification (failure creates a Teams message, may require having a shared email group, ideally this would also be visible to agency teams) (Not currently viable)
[ ] Email notification (the failure sends to a shared email group with both DSE and client staff CC'd) <- lower priority, Slack/Teams are preferred options.

summer-mothwood commented 3 weeks ago

@britt-allen I ran into this tutorial for using dbt webhooks for sending notifications to Teams when troubleshooting the Slack integration -- do we already use Zapier (or any other web automation tool that can work with webhooks)? https://docs.getdbt.com/guides/zapier-ms-teams?step=1

summer-mothwood commented 3 weeks ago

Looking more into dbt Cloud's webhook capabilities (and reviewing @ram-kishore-odi 's write up for Andrew on a somewhat similar issue) -- it looks like Microsoft Teams has some built-in functionality for subscribing to webhooks, too, so we might not even need the Zapier intermediate step!

britt-allen commented 3 weeks ago

We could send an email to a channel if we were not on the 365 Gov't plan...

britt-allen commented 3 weeks ago

This option could work, but I cannot install the required app because I had to make a request which I imagine goes to an Admin at GovOps. Lots of our ability to explore is blocked by ODI not having its own Microsoft license yet.

britt-allen commented 3 weeks ago

I have checked off Teams as not viable for now.

summer-mothwood commented 2 weeks ago

I'm looking into setting up Slack notification in dbt cloud. The first step is to set up Slack as an Account Integration in dbt Cloud. When I try to click through here

I get this error

Looking into it further (using these instructions https://docs.getdbt.com/docs/deploy/job-notifications#slack-notifications), I notice the docs say

You must be an account admin to configure Slack notifications in dbt Cloud.

I tried changing my license to an IT license, but still got the same error. @britt-allen do you know if one of us is an account admin on the dbt cloud account that has elevated privileges for that kind of thing? or should I put in a request ticket with dbt?

britt-allen commented 2 weeks ago

Let me give it a try!

britt-allen commented 2 weeks ago

I get the same error as well. I think this is because how IT has setup our dbt account, I'd go to our IT team first before opening a ticket with dbt @summer-mothwood

summer-mothwood commented 2 weeks ago

Thanks for looking into it @britt-allen !! I'll reach out to IT -- we'll need a system admin on the Slack side to complete the next steps anyway, so I'll ask them about that while I'm at it too

summer-mothwood commented 1 week ago

Put in a ticket to IT to integrate Slack and dbt for us.

summer-mothwood commented 1 week ago

I met with Kevin in IT today, and he was unable to configure Slack notifications / add dbt as an integrated app into Slack either. He's going to follow up with Chad and will notify me (in Slack, hah!) with updates. I mentioned again that our preference would be to enable notifications in Teams in order to make it easy for our clients who don't use Slack. Keeping the status here as blocked.

summer-mothwood commented 1 week ago

Chad ran into the same issue, so Kevin filed a ticket with dbt. Response from their team here, Kevin is working on implementing this morning:

Thank you for your patience. So it appears that there is an additional step that needs to be taken for Slack integration given your account is hosted on a Single Tenant instance. Would you be able to kindly do the following:

Create a Slack application here: https://api.slack.com/apps Once the above is completed, kindly provide the client ID and client secret. Please ensure to send the Client ID in plaintext and provide the Client secret using a self destructing service such as https://1ty.me/ for security purposes.

We're also looking into adding the additional requirement to our documentation here: https://docs.getdbt.com/docs/deploy/job-notifications#slack-notifications.

cagov / data-infrastructure

Investigate options for dbt failure notifications #415