DUP - Friday Bug Post-Mortem

[Dianadec]

Description:

[Description of the task]

https://medium.com/@pavan.kristipati/art-of-conducting-effective-blameless-postmortems-5f2b7765424c

Acceptance Criteria:

[Note: Use 'Given/When/Then' format if it makes sense to. Otherwise, a simple checklist that can be tested.]

Development Checklist:

• [ ] ...
• [ ] ...
• [ ] ...

Dependencies

• Blocked by
• Blocking

Relevant documentation as reference

Definition of Ready

• [ ] Acceptance criteria are included
• [ ] Wireframes are included (if applicable)
• [ ] Design / Solution is accepted by Product Owner (if applicable)
• [ ] Dependencies are identified (technical, business, regulatory/policy)
• [ ] Story has been estimated (under 13 pts)

Definition of Done

• In progress:
  • [ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
  • [ ] UI meets accessibility requirements
  • [ ] Unit tests are written
  • [ ] Work is traceable in GitHub
  • [ ] PR linked to ticket number
  • [ ] If needed/required - Dev adds flag/label to highlight any migration steps necessary prior to PROD deployment
• Code review:
  • [ ] Code is peer reviewed and has passed CI/CD tests
• QA:
  • [ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
• PO Review:
  • [ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
  • [ ] Reviewed and approved by Product Owner

Notes:

[meyerdarcie]

FYI @cameronpettit @Dianadec

Summary notes:

• Related to release of the hold pass timer feature, a bug was identified, which resulted in a negative number of passes showing as reserved. For front end users, this created unlimited availability, and QR codes could not be scanned in park.
• A number of actions were taken, including manually turning off passes, an interim fix to ensure passes were showing as booked in our database and admin portal, and reviewing system logs, GC Notify (email service) dashboard of sent emails, and cloud flare stats to identify how many passes were booked.
• Issue was resolved fully within 24hrs, with holds, reservations and cancellations all functioning correctly, and impact on the ground was minimal as rainy conditions resulted in a large amount of no-shows.
• QA documentation updated to reflect process improvement and lessons learned.

[meyerdarcie]

Revised with @cameronpettit 's input:

• Related to release of the hold pass timer feature, a bug was identified in the day use pass system, which resulted in a negative number of passes showing as reserved.
• The hold pass timer extracts passes from the total capacity when a user passes the Cloudflare turnstile, instead of when the pass booking is finalized. Extracted passes are in a 7 minute queue and in our code, the bug removed passes from the queue causing a fatal error - so regardless of whether the user completed their booking, the pass would be removed from the database after 7 minutes and capacity would be returned to the facility.
• For front end users, this created unlimited availability, and QR codes could not be scanned in park.
• A number of actions were taken, including manually turning off passes, an interim fix to ensure passes were showing as booked in our database and admin portal, and reviewing system logs, GC Notify (email service) dashboard of sent emails, and cloud flare stats to identify how many passes were booked.
• Despite passes being dropped from the database, we were able to reconstitute and return them to the database, so the issue was resolved almost immediately, and extra precautions were taken to prevent further passes from getting booked while the database was reconciled
• Issue was fully resolved within 24hrs, with holds, reservations and cancellations all functioning correctly. Our 'checked in' data shows that actual attendance was not significantly higher than regular capacity, so impact on the ground was minimal as rainy conditions resulted in a large amount of no-shows.
• QA documentation updated to reflect process improvement and lessons learned.

[manuji]

The updated documentation could be found here: https://apps.nrs.gov.bc.ca/int/confluence/display/BCPRS/Regression+Test+Plan+-+DUP

This test plan will be run prior to formal PROD deployments to make sure everything is working as expected.