DR | "Stack level too deep" error

[data-doge]

In Dec. 2023, we began to receive "SystemStackError: stack level too deep" errors on the Supplemental Claims form. See Sentry. This error is thrown, synchronously, at some during the creation/submission of a Supplemental Claim. Which makes it a submission-blocking error, but not a silent error. Interestingly, the error we see here is the result of our error handling code erroring. So we are unable to see the underlying reason for these errors. There is some discussion of this error here: https://dsva.slack.com/archives/C05FCEH2NRG/p1704901615183509.

An update (Feb 13), this error no longer appears to be occurring.

Another update: this error does appear to be occurring again.

Update 4/17/24: Refer to Derek's comment below for more details, but it looks like it's also happening in HLR

Tasks

• [ ] Fix our error handling code so that we can see the underlying errors blocking submissions.
• [ ] Confirm that these underlying errors, once exposed, result in notifications to our alerts channel in Slack.
• [ ] Check on Veterans impacted to see if they are still blocked.

[anniebtran]

It looks like we're seeing the error again — this specific veteran was able to submit successfully though (selected no evidence as a part of their form, so no evidence lost 👍 )

[anniebtran]

Another instance of it

[dfong-adh]

Recent occurrence

[anniebtran]

Seems like sometimes we don't get alerted by our monitors about this error occurring 😳 These two errors are in our dashboard but were not in our notifications channel: trace 1 and trace 2

[eileen-coforma]

Investigation in the works

[anniebtran]

Investigation notes:

• Seems to not have a consistent cause — it occurs for Supplemental Claim submissions with or without evidence, and has occurred for HLR, though it may have been a one-time error due to a poorly timed db migration.
• In some instances, the AppealSubmission record was created and InProgressForm destroyed, in others, only the AppealSubmission was created but the InProgressForm remained. Sometimes the code successfully completes the step to submit the form data to Lighthouse but if there's evidence, the code can error/blow up before we get to the step that uploads the evidence to Lighthouse and if the InProgressForm is deleted, we lose the data that would tie evidence back to the submission.

Suggestions/possible next steps:

• [x] Adding a rescue block to the PersonalInformationLog#create! call so that we can properly handle whatever error is being thrown there if there is one. It's possible that isn't exactly where the recursion is happening, but it might be a start.
  • ~We could double check and see if the PersonalInformationLogs are getting created, and that the error recursion exists somewhere else~ => Only found 1 record (not sure how often these records are purged), and it doesn't align with the user_uuids that have encountered this stack level problem.
• [x] Add a log before we enter the PersonalInformationLog related code to log the original exception that triggered this step to be taken

irb(main):019> PersonalInformationLog.where('error_class LIKE ?', "%V1::SupplementalClaimsController%").order(created_at: :desc).count
=> 1

• Might not be necessary with the work Derek is doing to persist form data until a submission is successful, but it could be valuable to think about how we could improve how we handle errors that occur in the middle of a transaction involving multiple steps/calls to the database

[eileen-coforma]

In code review. Poking platform.

[anniebtran]

PR was merged, working on validating but may take a few days as we don't see this error too frequently

[anniebtran]

Saw an instance of an error that leads to the stack level too deep:

Exception occurred while submitting Supplemental Claim: 1 validation error detected: Value at 'plaintext' failed to satisfy constraint: Member must have length less than or equal to 4096

Not sure what this means, but it's a start

[anniebtran]

Going to add the extra logging to NOD and HLR too since we saw a couple of stack level too deep errors for those forms as well 😩

[Mottie]

May open a new ticket to investigate errors from the logs

[anniebtran]

May have figured out what is possibly causing the stack level too deep error in Supplemental Claims, which is the form that most frequently encounters it, so I'll make a ticket to resolve it. But not sure what have caused it in NOD and HLR, but we can monitor for it now that we've added the extra logging

[anniebtran]

Haven't yet encountered the stack level too deep error from NOD and HLR since we've added the logging but upon inspecting their traces, they both happened around the same time and both share this error:

PG::UndefinedColumn: ERROR:  column form_submissions.in_progress_form_id does not exist
LINE 1: ...submissions" SET "in_progress_form_id" = $1 WHERE "form_subm...
                                                             ^

/usr/local/bundle/cache/ruby/3.3.0/gems/ddtrace-1.23.2/lib/datadog/tracing/contrib/pg/instrumentation.rb:35:in `exec_params': ERROR:  column form_submissions.in_progress_form_id does not exist
LINE 1: ...submissions" SET "in_progress_form_id" = $1 WHERE "form_subm...
                                                             ^
 (PG::UndefinedColumn)
    from /usr/local/bundle/cache/ruby/3.3.0/gems/ddtrace-1.23.2/lib/datadog/tracing/contrib/pg/instrumentation.rb:35:in `block in exec_params'
    ...

So going to look into this a bit and see if there are any other instances of this error 👀

[anniebtran]

Gonna close this since there's a new ticket to cover the fix for SC and we haven't seen the stack level too deep error in HLR and NOD occur again with the new logging (yet) — if we do encounter the error for those forms in the future, we can create a ticket as we'll be able to find more details around what may be causing it for them.