526ez Failed Claim Documentation

[kylesoskin]

Details

It has been decided that the 526ez historical claim remediation effort be transferred to the new 526 team(s).

This work has been on-going for around 1 year. The summary is, over the past 5+ years, there have been many cases of claims submitted that went into a blackhole. The VA is now remediating and reprocessing.

Terms

Submission IDs - The VA.gov internal database ID. Claim ID / VBMS ID - The internal ID in the system of record (VBMS). Participant ID - An identifier to Identify a Veteran. CMP / Central Mail Portal - A digital intake center at the VA for digital form submissions MAS / Mail Automation System - A team/system inside CMP that tries to automate processing of the form before they get into the CMP queue (where a human needs to look at them) Batches - Discrete subsets of claims, grouped by submission IDs.

Solutions

Backup Submission Path

The "band-aide" to "stop the bleeding". When submissions fail going directly to VBMS (via eVSS), they get slapped on a PDF and sent into Central Mail. Currently there are about 100-200 of these per week (that without the backup solution would otherwise have gone into the void). There are still some instances of these that fail too, that still go into the void, about 0-7 submissions per week.

Historical Remediation

All the historically failed claims (up to a point in time) were handed over to the VBA and evaluated. Of around 150k total failed submissions, they were deduplicated by the VBA into ~40k unique submissions (dedupped by veteran and contentions). These were originally being chunked up by which errors they had and which would be able to be loaded and when, however since so much time has passed the "batches" to be reprocessed have more or less become arbitrary. They have not yet actually loaded/turned any of them into claims yet. Also, all ones that are still failing, and all ones after the point in time of the original dedup that failed, will also need reproccessed somehow. Either manually through the backup submission process, or this historical process. They do not want to run all of them through the backup pdf process as manual human processing is expensive and it would overwhelm them to get 40k all at once. So they want to auto-establish the claims via a script, and are doing so in batches of claims.

They need the following for each "batch" to be able to create a claim systematically. This is the stuff we need to provide them.

PDF representations of the claims

• This has been done for ~50% of the 40k thus far
• There is a sidekiq job made to do this
• Something like this is what I used to pull these (all the code/classes to do this have already been made, but you need code to use them how you need)

batch = Sidekiq::Batch.new
throttle = Sidekiq::Limiter.concurrent("Form526BackupSubmission-#{Time.now.to_i}", 8, wait_timeout: 259_200,
                                                                                       lock_timeout: 120)
ids.each do |id|
  throttle.within_limit do
    batch.jobs do
      Sidekiq::Form526BackupSubmissionProcess::NonBreakeredForm526BackgroundLoader.perform_async(id)
    end
  end
end

This, for all IDs in ids queues a job to load each submission, generate all forms and ancillary forms, zip them as "#{id}.zip and upload them to S3. Then, you can provide ids to the VA people to pull them down via presigned S3 urls. I used a script like

concurency=24
count=0

function getaws {
  echo -n "."
  URL=$(aws s3 presign s3://$S3_BUCKET/$1.zip --expires-in 86400)
  echo "$1,\"${URL}\"" >> out.txt
}

for ID in $(cat ids_to_pull.txt); do 
  count=$((count+1))
  getaws $ID &
  if [ $count -eq $concurency ]; then
    wait
    count=0
  fi  
done

CSV For all claims in batch

• Snippet that can get this can be seen in this ticket
• This is used by a team to establish the claims via a script

Va Notify (they want to notify the Veterans, via email, after batch establishment)

• After they actually load/create the historical claims (which they have not yet done)
• They will provide a list of submission IDs that were successful
• You can use VA notify vets-api to send email to these veterans. This should be a one-time thing, so no automatic process is being built.
• I have a branch with this class, and was going to merge it, but have not yet...

# I have all the creds/keys and can provide if/when needed
class RemediationEmailer
  SECRET_KEY = ENV['SECRET_KEY']
  SERVICE_ID = ENV['SERVICE_ID']
  API_KEY    = "historical-526-emails-#{SERVICE_ID}-#{SECRET_KEY}"
  TEMPLATE_ID= ENV['TEMPLATE_ID'] 
  attr_accessor :returned_id
  def initialize(fname: nil, email: nil)
    @fname = fname
    @email = email
    @notify_client = VaNotify::Service.new(API_KEY)
  end
  def send_email!(form526_submission_id)
    begin
      if (@email.nil? || @fname.nil?)
        db_fname, db_email = get_email_and_first_name_from_submission_id(form526_submission_id) 
        @fname = db_fname if @fname.nil?
        @email = db_email if @email.nil? 
      end
      email_result = send_p
      @returned_id = email_result.id
      Rails.logger.info(
        "526 Historical Failure Email - success!",
        submission_id: form526_submission_id, 
        va_notify_id: @returned_id
      )
    rescue => e
      Rails.logger.error(
        "526 Historical Failure Email - FAILURE! #{e.message}.",
        backtrace: e.backtrace, 
        submission_id: form526_submission_id
      )
    end
  end
  private
  def send_p
    @notify_client.send_email(
      email_address: @email,
      template_id: TEMPLATE_ID,
      personalisation: {
        'first_name' => @fname
      }
    )
  end
  def get_email_and_first_name_from_submission_id(id)
    sub = Form526Submission.find(id)
    return [sub.auth_headers["va_eauth_firstName"], sub.form['form526']['form526']['veteran']['emailAddress']]
  end
end

# Emailer, but send all emails sent from it to provided email address (useful for testing)
# emailer = RemediationEmailer.new(email: "test@example.org")
ids = []
ids.each do |id|
  emailer = RemediationEmailer.new()
  emailer.send_email!(id)
end

Outstanding items

• There are still 0-7 going into the blackhole/void per week, due to validation errors, meaning we cannot get eVSS to generate a filled out 526 form with the data submitted. This should be fixed and be 0, so every submission goes the normal way or the backup way, no blackhole submissions.
• All of the historical claims need processed.
• All claims that are still failing, will also need processed.
• Probably other stuff.

[saderagsdale]

@kylesoskin @tblackwe @austinjprice Hey all - when is a good time to huddle on this work?

[austinjprice]

Monday or Tuesday is best as Thomas and will be out tomorrow

++ @SamStuckey

[saderagsdale]

Action items fro my chat with @austinjprice:

• Talk to devs Monday morning @tblackwe @kylesoskin @SamStuckey . Sam and Kyle will determine the best way to work on batch 3 and monitoring
• Update enablement team after