[07 NOV 2024] Monitor incident - Missing Pega status due to AWS degradation

[michaelclement]

Per playbook, opening ticket to track details of this DD alert in Slack.

Intake

• [X] PM (Bo) and DM (Andrea) have been notified
• [ ] PM and DM confirmation of ticket creation

Investigation

• [x] Verify that vets-api is receiving submits from vets-website
• [x] Verify that files are sent successfully to s3
• [x] Verify that the callback API is available
• [ ] Determine whether the issue arose due to a deployment, either by the IVC Forms team or the PEGA team.
  • [ ] IVC
  • [ ] If root cause has been identified as an issue in vets-website or vets-api, notify PM and DM to determine next stpes with PO
  • [ ] PEGA
  • [ ] If root cause has been identified as an issue outside of vets-website and vets-api, work with the PM, DM, and PO to determine contacts and next steps.

Resolution

• [x] Acknowledge monitoring alerts in slack
• [ ] IVC Forms engineering will provide updates on development and deployment of a solution and its status in each environment
• [ ] IVC Forms engineering will notify PM and DM when resolved
• [ ] IVC Forms engineering will monitor outstanding missing statuses and ensure they are cleared appropriately. No statuses should be deleted or cleared without agreement from PO and or PM
• [ ] If traffic to the site has been altered, PO or PM will determine when to return to previous levels.

[michaelclement]

Reached out to Cindy on Teams at 1:08pm CT, she had this info:

I am receiving the documents but AWS is being really slow. My logs files are not showing up until 10 minutes after the trigger was processed. This is all environments inside the VAEC

[michaelclement]

VAEC = Veterans Affairs Enterprise Cloud

[michaelclement]

This is sort've an odd case - it's not a problem with our system or Pega, but rather looks to be an AWS slowdown. For the time being, will simply monitor the situation. After things return to normal (which they hopefully should soon) we should:

• Identify the IDs of submissions without Pega statuses and verify with Cindy that they were indeed processed.
• Manually update the Pega statuses in the database
• Clear the DD alerts