Closed kjduensing closed 1 month ago
Including @mchelen-gov for prioritization help
+1 for this feature request.
For Public Websites, we use Sentry for error monitoring for the Forms product which relies on vets-api: Find a Form error report. We have also configured Alerts: Forms errors warning / critical notifications, but they don't do anything. We don't receive a notification or email. Would be great if we could. (cc @wesrowe )
Hi @little-oddball & team - how can I check the status on this item? Also who do I need to work with to prioritize this? 🙇
Hi DevOps team. I'm checking in on the status for this item. We have production issues and we are trying to address (in part) by configuring Sentry/Slack integration - what can we do to prioritize (or get a status update) on this work? I'm hoping this is as simple as someone pressing the big "Enable Sentry integration" button, but please let me know if there is more work involved that we can assist with.
@mchelen @mchelen-gov Hi Mike - back in November with PTEMS Platform transition, I had heard from Ian Hundere (now deactivated) that Sentry didn't have an owner. In the new world of Platform tech teams, can you help us identify which team owns Sentry?
Out sick but am addressing this on my return.
New info per @pjhill (summarized, please correct me if I got anything wrong):
Peter brought this request to the Platform DevOps Community of Practice (CoP). From that, they determined the following needs to be clarified before working on this request:
From my perspective, Sentry holds an important place in debugging, error tracing, and alerting. I see Sentry and Datadog performing 2 different functions.
Functionality aside, in addition to invalidating our tooling, processes, and documentation, the Sentry -> Datadog migration might represent actual code changes on our side, which slows down delivery of features.
Also, the Platform's customers are a little bit "migration weary" right now. There are several non-optional migrations (EKS & EVSS sunsetting) many VFS teams are having to implement.
Obviously, I prefer getting Sentry/Slack integration working :). But these are just my thoughts, more user research is needed by the Platform to see if the themes above resonate throughout VFS teams.
Thank you all for the information! To clarify for myself, my immediate next steps:
Related to the Issue #1, what comes up are follow-up questions below:
Related to Issue #2:
Will add answers/more notes shortly...
Platform team roll call can be found under subtitle Digital Modernization: https://docs.google.com/document/d/1QmqnVRxLbUEw2Bxc9xmMMhdihjX4SkSgCDcXofkktnc/edit?pli=1
Platform VA Docs doesn't list Sentry as currently in use in the Health and Monitoring Rubric (DataDog is listed): https://depo-platform-documentation.scrollhelp.site/analytics-monitoring/track-your-kpis.
Slack is filled with references to active Sentry usage, it's actively used and referenced. Obtained entry to VSP's Sentry instance.
This Sentry doc goes over how to add and configure Sentry alerts to Slack and is linked in the threads referenced above: https://docs.sentry.io/product/integrations/notification-incidents/slack/
The history linked in the description of this ticket shows ^these instructions not working.
Tracking down any possible Sentry owners/admins.
Re alerting that's already expected to exist, this doc exists on the vfs side: https://vfs.atlassian.net/wiki/spaces/pilot/pages/590774316/Tracking+application+errors+with+Sentry
Will reach out to authors to see if they are, or know, the Sentry owner/admin/SME.
Determined that existing alerting workflow doesn't currently work: https://vfs.atlassian.net/wiki/spaces/pilot/pages/590774316/Tracking+application+errors+with+Sentry
Author is from content but they may know identity of information owner/s?
Spoke with Kevin and Peter to clarify questions raised in the ticket. Kevin confirmed that the above workflow still needs a Sentry/Slack integration to work; he's attempted using that workflow before and it didn't yield Slack/Sentry messages.
Also worth noting that at least two Slack rooms currently do receive Sentry notifications, but they may be accessing a different Slack instance, rather than sentry.vfs.va.gov.
Priorities for tomorrow are ensuring the following questions are finished before diving more into integration details:
Determine and document who owns Sentry (if anyone—according to the epic, this tool may not have an owner anymore.)
a. If there is no owner, determine if there's a process in place for assigning ownership.
b. Document ownership
Determine and document the mechanisms for Sentry/Slack alerting in place (forsaken and active versions)
a. Could we piggyback on any or do any mirror the ask in the epic?
b. What's different between the ask in the ticket and the existing mechanisms?
c. Why do some Slack rooms already receive regular Sentry alerts if a Slack integration isn't already activated?
1. Are they using a different Sentry instance?
2. Determine how to confirm which instance their accessing?
a. Check out: #appeals-app-alerts && #appeals-queue-alerts
As DataDog migration is no longer expected to be used as part of Slack/Sentry workflow, the next areas of completion likely include:
Need to confirm safety and feasibility with devops COP of possible integration plan.
Hi @kjduensing, we are still in progress on this - we had tabled it shortly while we've been focused on other issues, but are continuing to work on it this sprint.
@gia-lexa any updates here? We have active incidents every now and then (example) where this push from Sentry would be really really helpful.
Hi @jilladams, absolutely. Most recently, the proposed implementation plan was approved by DevOps COP—they are the owners of the Slack and Sentry configurations.
We're now working through this in-progress ticket, which is a collaboration with DevOps COP to configure and run a closed test of the proposed implementation. DevOps has assigned that work to a member of their team. And I usually update that ticket's status on Thursdays after I attend the DevOps COP meeting.
A note about the closed test—once configured, it will run from 1- 2 weeks while we monitor the output to ensure we're adhering to best practices.
Once we successfully complete that test, we'll flip on a configuration accessible to the appropriate members of vfs as indicated by this upcoming ticket.
(There will also be a final ticket for documentation.)
That's the current overview, I hope it's helpful. Nonetheless, please don't hesitate to let me know if you have any questions about anything I've written here, or if there are any further details that would be helpful for me to share.
Hi @gia-lexa thanks for the detailed status from a while back! Is it safe to say this ticket can be closed? Doing a bit of cleanup and closing some loops. Thanks!
Hi @gia-lexa and @little-oddball - just following up on my previous comment. Is it safe to close this ticket? Thanks!
Hi Nicole! Apologies, I missed your first email!
Before I rolled off this team, I was told that a workaround—which included using Datadog—had been implemented and that this ticket would subsequently be closed.
I hope that helps, but please let me know if I can provide you any further details! (Also if it's easier, happy to chat on slack, @gia). Thanks!
Thank you @gia-lexa! That does help!
Hi @little-oddball - would you mind closing this ticket if the request is considered complete? Thanks!
Hi @little-oddball - any update on this item?
Hi @little-oddball - any update on this item?
@gopixelsgo - I wasn't assigned to the issue so haven't been tracking it at all.
Sentry is an item Platform is working to deprecate so that aspect of this ticket doesn't make sense to invest in moving forward. As far as DD, it should already be working w/ Slack Integration.
I would suggest closing this ticket.
Describe the problem
Monitoring and alerting are a central component of a mature product team. Though it's no fault of Platform teams, we have been significantly impacted by a lack of autonomy and unwieldy nature of the only available alerting process in Sentry.
Alerting on VFS teams is currently limited in the following ways:
Who will benefit
It's clear that all VFS teams will benefit by having control over their alerting system. However, Platform teams will also see a decrease in requests to configure alerts for VFS teams, reducing the burden on platform support.
Describe your idea
The simplest idea we have is to enable Slack integration in the current
http://sentry.vfs.va.gov/
Sentry instance (VSP
organization). There may be technical or procedural issues with this approach, however, enabling Slack integration will allow VFS teams to configure their own slack integrations with dedicated channels to proactively handle any incidents that occur.If Slack integration is unfeasible due to level of effort or a technical/procedural blocker, it looks like there may be some history of using webhooks to sent errors to DataDog and then having DataDog take care of the alerting. However, VFS teams do not have write access to DD, so this would require platform support as well.
Finally, email is an acceptable alerting mechanism, but there are drawbacks vs Slack. If we could figure out how to re-enable sending alerts via email from Sentry, this would be a step in the right direction.
Provide evidence
https://dsva.slack.com/archives/CBU0KDSB1/p1676059425011579
https://dsva.slack.com/archives/CBU0KDSB1/p1678122264462019
https://dsva.slack.com/archives/CBU0KDSB1/p1678721148949179
https://dsva.slack.com/archives/CBU0KDSB1/p1668707111130899
https://dsva.slack.com/archives/CBU0KDSB1/p1659448501942089
https://github.com/department-of-veterans-affairs/va.gov-cms/issues/4696
https://dsva.slack.com/archives/CBU0KDSB1/p1642535108486300?thread_ts=1642187323.408700&cid=CBU0KDSB1
Platform Mission
Other:
No response