search

bcgov / digital-journeys

PSA Forms System
https://bcgov.github.io/digital-journeys/
Apache License 2.0
8 stars 7 forks source link

SL Review: Investigation on the production issues #1373

Closed iman-jamali-fw closed 8 months ago

iman-jamali-fw commented 1 year ago

Document all production issues that happened and lessons learned.

iman-jamali-fw commented 1 year ago

updated on Oct 11:

I've conducted a thorough investigation into a discrepancy issue related to SL (Service Level) submissions. This investigation involved comparing all the SL submissions from various data sources, including Formio MongoDB, API PostgresDB, Camunda PostgresDB, and ODS. The comparison focused on key elements such as creation and last update timestamps, application status, and critical content. Here's what I discovered:

There were seven cases where the data in ODS did not reflect the latest changes from DGJ. To resolve this, I manually updated these seven cases in ODS to include the latest information from DGJ.

The seven impacted submissions can be categorized as follows with link and applicationIds:

  1. Three submissions were made right after an OpenShift outage and when DGJ was coming back online on October 4th. It appears that DGJ wasn't fully operational when users made these updates. Camunda might not have been fully functional, and as a result, the data wasn't sent to ODS. applicationId: 60776 link: https://digital-journeys.apps.silver.devops.gov.bc.ca/form/64754420faabe6b3fc9a5abb/submission/64fa24c0889f59029db2bbea

applicationId: 60423 link: https://digital-journeys.apps.silver.devops.gov.bc.ca/form/64754420faabe6b3fc9a5abb/submission/65033f19889f59029db5bf0c

applicationId: 60428 link: https://digital-journeys.apps.silver.devops.gov.bc.ca/form/64754420faabe6b3fc9a5abb/submission/65033e2c889f59029db5bd69

  1. Another group of three submissions couldn't be updated in ODS due to the presence of special characters in the form fields (quotes and dashes) use should have copied the text from a text editor. It's still uncertain whether this issue was at the ODS level or if there was a problem with the HTTP client responsible for sending the data from DGJ. (@Bhumin Patel and @Mike Shi can comment as there was a similar issue on another form before). The special characters in question were \u2019, \u2013, and \u2018. I had to manually replace them with alternative ASCII characters and then resend the data to ODS. applicationId: 62577 link: https://digital-journeys.apps.silver.devops.gov.bc.ca/form/64754420faabe6b3fc9a5abb/submission/6504ec2a889f59029db716c1

applicationId: 54092 link: https://digital-journeys.apps.silver.devops.gov.bc.ca/form/64754420faabe6b3fc9a5abb/submission/64b16b55e0e06917f21423f9

applicationId: 58225 link: https://digital-journeys.apps.silver.devops.gov.bc.ca/form/64754420faabe6b3fc9a5abb/submission/64e55093e0e06917f21c5b1a

  1. For the last impacted submission, I haven't yet determined the cause of the issue. This is an older instance submitted in June, and my suspicion is that changes to the data structure and workflow might be responsible for the failure to update in ODS. applicationId: 51497 link: https://digital-journeys.apps.silver.devops.gov.bc.ca/form/64754420faabe6b3fc9a5abb/submission/649b5345e0e06917f20f4520
Stella-Archer commented 8 months ago

1/ load balancing issue - happened between submission need to investigate further - keep an eye on telework 2/ Character replacement script added to cold and flu 3/ Writing/updating in workflow processing of data is lost in between data

Stella-Archer commented 8 months ago

This is a mentoring tix not explore so closing ticket

fazil-ey commented 8 months ago

closing as stella suggested above