search

getsentry / sentry

Developer-first error tracking and performance monitoring
https://sentry.io
Other
38.6k stars 4.13k forks source link

Sentry MSTeams Integration not reporting all alert notifications #77681

Open cah-michel-bitar opened 2 days ago

cah-michel-bitar commented 2 days ago

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

We recently upgraded from 23.3.1 to 24.8.0 and we had to reconfigure Microsoft Teams integration, and the issue started to happen.

We have alerts configured to send notifications to Microsoft Teams when a new issue is created with certain message content filters.

Action Interval is 30mins.

The specific event expected to be reported to Microsoft Teams shows under the Alert as triggering it, eg: Image

Expected Result

We expected to receive an MS Teams message notification.

Actual Result

We did not receive a notification, only for that specific event id. MS Teams notifications were being received +-1h time's notifications were received. Just this one was not. This is happening occasionally now and seems to be quite random. I was not able to find any commonality between the specific events not being reported (although triggering the alert)

The only logs related to MS Teams I was able to find in the web container are the following:

web-1  | 2024-09-12T18:13:02.258835526Z 18:13:02 [INFO] sentry.integrations.msteams.webhooks: sentry.integrations.msteams.webhook (request_data={'type': 'message', 'timestamp': '2024-09-12T18:12:57.929Z', 'localTimestamp': '2024-09-12T14:12:57.929-04:00', 'id': 'f:af304f7e-6e8d-30c5-9550-2f4d820b0a2d', 'channelId': 'msteams', 'serviceUrl': 'https://smba.trafficmanager.net/amer/', 'from': {'id': '29:1h62x77I92gxZHXAn9WZACSU3xwyGpSl9nw8OROHvMd*****_3BCUGNu9Ulyyz73alw2nlX7MEB93y24_9wOO2Q', 'name': '*****', 'aadObjectId': 'c1c384ab-fbf1-4aff-9e4b-4c2824f*****'}, 'conversation': {'isGroup': True, 'conversationType': 'channel', 'tenantId': 'a86dc674-e6a2-4c25-b4ba-e36c313*****', 'id': '19:aace3adaa78044cb9ae7f23336a*****@thread.skype;messageid=1726155083650', 'name': 'Channel Name'}, 'recipient': {'id': '28:c32f9241-2c75-4524-a383-fd47fd6*****', 'name': 'Sentry msteams bot'}, 'entities': [{'locale': 'en-US', 'country': 'US', 'platform': 'Windows', 'timezone': 'America/New_York', 'type': 'clientInfo'}], 'channelData': {'channel': {'id': '19:aace3adaa78044cb9ae7f23336a*****@thread.skype'}, 'team': {'id': '19:9cfc0442beea4a7f96f01cf0ce7*****@thread.skype'}, 'tenant': {'id': 'a86dc674-e6a2-4c25-b4ba-e36c313*****'}, 'source': {'name': 'message'}, 'legacy': {'replyToId': '1:1tH-MfSjHL0K55EI1f56Y3XJCydI0AN4odBOKn*****'}}, 'replyToId': '1726155083650', 'value': {'payload': {'actionType': '1', 'groupId': 289, 'eventId': '5ffc6c60c8ee4d8da0e015558851a0c4', 'rules': [35], 'integrationId': 7}, 'resolveInput': 'resolved'}, 'locale': 'en-US', 'localTimezone': 'America/New_York'})
web-1  | 2024-09-12T18:13:11.014849387Z 18:13:11 [INFO] sentry.integrations.msteams.webhooks: sentry.integrations.msteams.webhook (request_data={'type': 'message', 'timestamp': '2024-09-12T18:13:06.949Z', 'localTimestamp': '2024-09-12T14:13:06.949-04:00', 'id': 'f:658f5e70-bbcc-e1e7-c6fb-d0bc6ca4fb89', 'channelId': 'msteams', 'serviceUrl': 'https://smba.trafficmanager.net/amer/', 'from': {'id': '29:1h62x77I92gxZHXAn9WZACSU3xwyGpSl9nw8OROHvMd*****_3BCUGNu9Ulyyz73alw2nlX7MEB93y24_9wOO2Q', 'name': '*****', 'aadObjectId': 'c1c384ab-fbf1-4aff-9e4b-4c2824f*****'}, 'conversation': {'isGroup': True, 'conversationType': 'channel', 'tenantId': 'a86dc674-e6a2-4c25-b4ba-e36c313*****', 'id': '19:aace3adaa78044cb9ae7f23336a*****@thread.skype;messageid=1726156903285', 'name': 'Channel Name'}, 'recipient': {'id': '28:c32f9241-2c75-4524-a383-fd47fd6*****', 'name': 'Sentry msteams bot'}, 'entities': [{'locale': 'en-US', 'country': 'US', 'platform': 'Windows', 'timezone': 'America/New_York', 'type': 'clientInfo'}], 'channelData': {'channel': {'id': '19:aace3adaa78044cb9ae7f23336a*****@thread.skype'}, 'team': {'id': '19:9cfc0442beea4a7f96f01cf0ce7*****@thread.skype'}, 'tenant': {'id': 'a86dc674-e6a2-4c25-b4ba-e36c313*****'}, 'source': {'name': 'message'}, 'legacy': {'replyToId': '1:1g1APO8NMTGr1NTXiXdv1JFqiWCMGfNwMjL2iB*****'}}, 'replyToId': '1726156903285', 'value': {'payload': {'actionType': '1', 'groupId': 290, 'eventId': '6cc8e54b890f46a5bf32277920fedd99', 'rules': [35], 'integrationId': 7}, 'resolveInput': 'resolved'}, 'locale': 'en-US', 'localTimezone': 'America/New_York'})
web-1  | 2024-09-12T19:17:04.409053933Z 19:17:04 [INFO] sentry.integrations.msteams.webhooks: sentry.integrations.msteams.webhook (request_data={'type': 'message', 'timestamp': '2024-09-12T19:17:01.666Z', 'localTimestamp': '2024-09-12T15:17:01.666-04:00', 'id': 'f:a8e905e0-c946-07d9-418d-0b8ff08cc4c7', 'channelId': 'msteams', 'serviceUrl': 'https://smba.trafficmanager.net/amer/', 'from': {'id': '29:1h62x77I92gxZHXAn9WZACSU3xwyGpSl9nw8OROHvMd*****_3BCUGNu9Ulyyz73alw2nlX7MEB93y24_9wOO2Q', 'name': '*****', 'aadObjectId': 'c1c384ab-fbf1-4aff-9e4b-4c2824f*****'}, 'conversation': {'isGroup': True, 'conversationType': 'channel', 'tenantId': 'a86dc674-e6a2-4c25-b4ba-e36c313*****', 'id': '19:aace3adaa78044cb9ae7f23336a*****@thread.skype;messageid=1726168329349', 'name': 'Channel Name'}, 'recipient': {'id': '28:c32f9241-2c75-4524-a383-fd47fd6*****', 'name': 'Sentry msteams bot'}, 'entities': [{'locale': 'en-US', 'country': 'US', 'platform': 'Windows', 'timezone': 'America/New_York', 'type': 'clientInfo'}], 'channelData': {'channel': {'id': '19:aace3adaa78044cb9ae7f23336a*****@thread.skype'}, 'team': {'id': '19:9cfc0442beea4a7f96f01cf0ce7*****@thread.skype'}, 'tenant': {'id': 'a86dc674-e6a2-4c25-b4ba-e36c313*****'}, 'source': {'name': 'message'}, 'legacy': {'replyToId': '1:1nRtx-MrjnQrqG43GWy52y1E8qNjYkG6z0UifT*****'}}, 'replyToId': '1726168329349', 'value': {'payload': {'actionType': '1', 'groupId': 302, 'eventId': '1d90e0589f124a5bb393779d76542323', 'rules': [60], 'integrationId': 7}, 'resolveInput': 'resolved'}, 'locale': 'en-US', 'localTimezone': 'America/New_York'})
web-1  | 2024-09-13T20:14:45.439021500Z 20:14:45 [INFO] sentry.integrations.msteams.webhooks: sentry.integrations.msteams.webhook (request_data={'type': 'message', 'timestamp': '2024-09-13T20:14:40.809Z', 'localTimestamp': '2024-09-13T16:14:40.809-04:00', 'id': 'f:e887eb7f-420d-81cb-278d-ed0b8b985df7', 'channelId': 'msteams', 'serviceUrl': 'https://smba.trafficmanager.net/amer/', 'from': {'id': '29:1h62x77I92gxZHXAn9WZACSU3xwyGpSl9nw8OROHvMd*****_3BCUGNu9Ulyyz73alw2nlX7MEB93y24_9wOO2Q', 'name': '*****', 'aadObjectId': 'c1c384ab-fbf1-4aff-9e4b-4c2824f*****'}, 'conversation': {'isGroup': True, 'conversationType': 'channel', 'tenantId': 'a86dc674-e6a2-4c25-b4ba-e36c313*****', 'id': '19:aace3adaa78044cb9ae7f23336a*****@thread.skype;messageid=1726244767674', 'name': 'Channel Name'}, 'recipient': {'id': '28:c32f9241-2c75-4524-a383-fd47fd6*****', 'name': 'Sentry msteams bot'}, 'entities': [{'locale': 'en-US', 'country': 'US', 'platform': 'Windows', 'timezone': 'America/New_York', 'type': 'clientInfo'}], 'channelData': {'channel': {'id': '19:aace3adaa78044cb9ae7f23336a*****@thread.skype'}, 'team': {'id': '19:9cfc0442beea4a7f96f01cf0ce7*****@thread.skype'}, 'tenant': {'id': 'a86dc674-e6a2-4c25-b4ba-e36c313*****'}, 'source': {'name': 'message'}, 'legacy': {'replyToId': '1:1Kq3cDiJ5RR64WCrXL6rXGP49vQKTOgQXHJ1_N*****'}}, 'replyToId': '1726244767674', 'value': {'payload': {'actionType': '1', 'groupId': 380, 'eventId': '9cf848be88c44f119914f620326dbe25', 'rules': [60], 'integrationId': 7}, 'resolveInput': 'resolved'}, 'locale': 'en-US', 'localTimezone': 'America/New_York'})

These are the only logs related to MS Teams that I was able to find. These appear to be the logs when the issue status is updated to "Resolved". The event in question is '1d90e0589f124a5bb393779d76542323' After manually finding it out on the UI, and resolving it from Sentry, it seems like there was a log mentioning that the issue got updated to "Resolved". However, no MS Teams message was found.

Has this happened before? Is there a way to debug this further? Any entries in the DB might help find the root cause?

Product Area

Settings - Integrations

Link

No response

DSN

No response

Version

24.8.0

getsantry[bot] commented 2 days ago

Assigning to @getsentry/support for routing ⏲️

getsantry[bot] commented 1 day ago

Routing to @getsentry/product-owners-settings-integrations for triage ⏲️

sentaur-athena commented 11 hours ago

Taking this back to team and will update soon.

RyanSkonnord commented 10 hours ago

Thank you for the report. We haven’t observed this happening on our production environment. A failure to send a notification should certainly produce an error log, and without one it’s unfortunately going to be difficult to investigate.

One possible problem we’re aware of is that, when we fire a message to Teams and it times out, it can sometimes fail opaquely. Just speculating, but a possible cause could be if there’s especially high latency between your Sentry server and the Teams API.

We’re actively working on improving our logging around messaging platform integrations, so if there is an error condition not being properly logged, hopefully we’ll find it soon. Thanks again for reaching out.