[10/18/2024 - 1:16AM ET] Monitor incident - Missing PEGA status

[michaelclement]

Per playbook, opening ticket to track details of these DD monitor alerts (and followup).

Intake

• [x] PM (Bo) and DM (Andrea) have been notified
• [x] PM and DM confirmation of ticket creation

Investigation

• [x] Verify that vets-api is receiving submits from vets-website
• [ ] Verify that files are sent successfully to s3
• [x] Verify that the callback API is available
• [ ] Determine whether the issue arose due to a deployment, either by the IVC Forms team or the PEGA team.
  • [x] IVC
  • [x] If root cause has been identified as an issue in vets-website or vets-api, notify PM and DM to determine next stpes with PO
  • [ ] PEGA
  • [ ] If root cause has been identified as an issue outside of vets-website and vets-api, work with the PM, DM, and PO to determine contacts and next steps.

Resolution

• [x] Acknowledge monitoring alerts in slack
• [ ] IVC Forms engineering will provide updates on development and deployment of a solution and its status in each environment
• [ ] IVC Forms engineering will notify PM and DM when resolved
• [ ] IVC Forms engineering will monitor outstanding missing statuses and ensure they are cleared appropriately. No statuses should be deleted or cleared without agreement from PO and or PM
• [ ] If traffic to the site has been altered, PO or PM will determine when to return to previous levels.

[michaelclement]

Notes for changes we want to make as a result of this:

• Research the ability to make a DD monitor track if users are seeing error screens on the FE
• Send full form submission JSON to Pega so we can recreate PDF or test as necessary

[michaelclement]

Relevant task on board for storing all JSON @cloudmagic80

[andrea-merrill]

Adding the Watch Tower Ticket link. We'll want to circle back to this once this is resolved.

[michaelclement]

DSVA thread where most communication has been happening

[michaelclement]

Top level ticket - more individual tickets to come, but this represents the DD goals in one place.

[michaelclement]

Engineers will be having a meeting at 12CT today to discuss moving forward with DD improvements.

[cloudmagic80]

Notes for backend: Create a ticket for save parsed json data into PEGA S3. https://app.zenhub.com/workspaces/ivc-forms-652da2d3f0ae4c0016bfb109/issues/gh/department-of-veterans-affairs/va.gov-team/95328