freeheeling
commented
10 months ago @austinjprice After a thorough review of all readily available logs within Datadog, the following summary of 4142 submission failures, going back to 8/30/2022, were identified:
- 40 submissions with invalid character(s) detected (the last instance occurring 12/5/23)
- 42 submissions with an invalid zipCode value (the last instance occurring 11/12/23)
- 407 submissions with an invalid NumberPages value (all instances occurring between 2/4/23 and 2/6/23)
Submission IDs and example values have been obtained for each error and shared with Emily Theis.
Causes for the first two failures listed may be attributed to incomplete/incorrect metadata validation documentation supplied by the EMMS team. Having been provided with the corrected regex validations, updates to normalize the respective metadata fields to match expectations have been submitted for review to prevent any future failures associated with the corresponding values.
The cause of the last listed failure, related to a discrepancy between a numberPages value sent in the metadata and the actual number of pages of the main document, remains unsolved. Without knowing how to determine the actual number of pages, I remain blocked in my attempt to compare the two numbers. It is suspicious, however, that the only instances where this type of error occurred happened during a 3-day period at the beginning of February, and an error hasn't been raised since.
To estimate the extent of failures, relative to the total number of forms submitted for processing, the number of 526 submissions containing form 4142 data was calculated to be 151,935 during the date range of 12/1/22 to 12/27/23, which equates to approximately 0.3% of all forms for that time period.
A number of discoveries are worth mentioning as a result of the work that went into retrieving this data:
- Log data within Datadog is only available on the dashboard as far back as 1 year; anything prior to a year is archived and has to be rehydrated at a cost.
- Research performed by Sam Raudabaugh concluded that:
- general Rails server logs, queryable in Datadog, only go back as far as ~August 2022
- server logs for ~June 2021 to ~August 2022 should be in Loki, but querying isn't currently possible
- CloudWatch should have server logs pre-June 2021
- The rehydrated logs for data used in this investigation, accessible within Datadog's Log Explorer, are only available until January 18, 2024.
It was discovered that some percentage of 4142 forms are not successfully reaching their final destination when being generated in the 526 pathway.
First step to resolving this issue is to determine how often this is failure, what type of failures are occurring and create a list of submissions that were impacted.
Acceptance criteria: