Send email on successful 526ez submission

[anushyaprasad]

As a veteran when submitting disability compensation form I want to receive email confirmation of my submission So that I have a record of the submission with reference number.

Checklist:

• [x] add new feature toggle (https://department-of-veterans-affairs.github.io/veteran-facing-services-tools/platform/tools/feature-toggles)
• [x] add user to Flipper admin list in settings.yml
• [x] create a new sidekiq job to send email using notifications-ruby-client (example: DirectDepositEmailJob)
• [x] initialize client with settings from config (api key)
• [x] retrieve email from the form JSON.parse(form_526_submission.form_json)[“form526”][“form526"][“veteran”][“emailAddress”]
• [x] trigger the job after successful submission (perform_ancillary_jobs (Form526Submission#perform_ancillary_jobs)
• [x] verify error handling for sidekiq job
  • pass submission_id
  • example error when template id is not available on notifications api side
• [x] add statsd stuff and possibly enhance logging for this case
• [ ] ~double check with Anna and Sarah-Jaine that ...[veteran][emailAddress] is a required field, and that we should use submission_id and not submitted_claim_id~ - not relevant because [veteran][emailAddress] is the only email we have access to, and the claim ID is not really used in this story
• [x] find a way of triggering the confirmation email in a non-batch job in app/models/form526_submission
• [x] ask operations to get the secret key in credstash for the development/review instance
• [x] make a PR to this file for the review instance and this file for the dev instance and add configuration for the template ID and the lookup(credstash, path-to-secret-key) bits
• [x] make a PR in vets-api for our changes (which should deploy onto the review instance)
• [x] give it a try on the review instance

To revisit:

• [ ] add test with mocking to check that disabled features return false for the first value (feature_toggles_controller_spec.rb)

Assumption:

• Includes adding the feature toggle.
• Handling system errors will be done as a part of another story #86

Open questions:

• [x] Where to store template id?
• Storing template id in settings.yml, under vanotify.template_id.form526_confirmation_email. This will be overridden by this file for the review instance and this file for the dev instance

Additional useful context:

• Difference between v0 and v1 controllers:

app/controllers/v0/disability_compensation_forms_controller.rb is the controller used when a veteran is logged into VA.gov and wants to submit a claim. This is the code that our contract deals with . modules/claims_api/app/controllers/claims_api/v1/forms/disability_compensation_controller.rb is the api called by a VSO working to submit a claim on behalf of a veteran. This code is written by the Lighthouse team (separate contract).

• Email address in the form:

  data gets populated from VA Profile, but user can override them only for this specific application

• Developers working on disability form (invited to our #notifications-engine):

  • Anna Carey
  • Sarah-Jaine Szekeresh

Learnings:

• Tests take a long time to run.
• There is also no task to run specific tests within docker. We could look into writing a specific task in the makefile.
• Flipper is used for feature toggles. We were able to disable the feature in the dev environment, but we had trouble passing the appropriate gate (boolean) so it doesn't currently show up.
• Set-up issues were related to docker.
• Google Analytics integrated for DirectDepositEmailJob - would we want to consider similar integration of Google Analytics for our job in the future?
• vets-api submits the saved claim to EVSS, which returns an ID that we save as submitted_claim_id - this is seemingly used in generating some PDFs, so it's probably the most meaningful ID that we should be using in email personalisation going forward. Should still confirm this when we get to doing personalisation.

[ffafara-tw]

An example of sidekiq job that sends emails: app/workers/direct_deposit_email_job.rb

Used here: app/controllers/v0/ppiu_controller.rb#send_confirmation_email

[QaysarA]

We should discuss the scenarios that trigger the submission email on va.gov (I found some documentation around two success confirmation messages and other error messages - but not sure if they are still applicable)

[ffafara-tw]

UI polling logic is here: vets-website/src/applications/disability-benefits/526EZ/components/ConfirmationPoll.jsx Confirmation page: vets-website/src/applications/disability-benefits/526EZ/containers/ConfirmationPage.jsx

[philherbert]

Taylor Skinner has added configuration for VANotify to the fwdproxy in this commit https://github.com/department-of-veterans-affairs/devops/commit/b36345add9d58148aaf0c5dfb388e4a642717cd0#diff-f2be7bb8d7e38cd735cc6f4af56d0e24

[philherbert]

A recap of the current issues:

1. send_form526_confirmation_email fails because of a JSON parsing error:

   2020-06-23 20:10:50.225655 W [20:47342534670460] Sidekiq -- TypeError: no implicit conversion of nil into String
   2020-06-23 20:10:50.225673 W [20:47342534670460] Sidekiq -- /usr/local/bundle/gems/json-2.3.0/lib/json/common.rb:156:in `initialize'
   /usr/local/bundle/gems/json-2.3.0/lib/json/common.rb:156:in `new'
   /usr/local/bundle/gems/json-2.3.0/lib/json/common.rb:156:in `parse'
   /srv/vets-api/src/app/models/form526_submission.rb:154:in `send_form526_confirmation_email'
   /srv/vets-api/src/app/models/form526_submission.rb:128:in `workflow_complete_handler'
   /usr/local/bundle/gems/sidekiq-pro-5.0.0/lib/sidekiq/batch/callback.rb:37:in `execute_callback'

   We're going to do some more digging into this tomorrow, because the form_json is definitely present in the database.

2. Form526ConfirmationEmailJob.perform_async fails because of an issue with OpenSSL - the configuration for the review instance has a client_url that starts with https, which the instance can't access. Raised department-of-veterans-affairs/devops#7159 to fix.

[philherbert]

PR department-of-veterans-affairs/vets-api#4426 requested review!

[philherbert]

Waiting on review from Anna, we'll give it a day and then move it

[lingtran]

Awaiting verification of feature success in dev environment. We currently can't seem to successfully submit a claim. Started a thread in platform support seeking help. https://dsva.slack.com/archives/CBU0KDSB1/p1593456555482200

[marisahoenig]

Staging configuration and staging API key (pointing to our dev environment) has been added, so now, we should be able to test in the vets-api staging environment (after we turn the flipper on!).

[marisahoenig]

Issues we're currently having: review instance here

1. When we added the new load balancers, we did not include the port :80 in the service addresses, so we added these in, merged the PR, and it should be deployed momentarily.

   • The error we saw was service unavailable (503) for the review instance.
   • Tested with email vets.gov.user+15@gmail.com

2. In review instance, we received an error for certain users that reached max EP codes. We think we need to reset the counts for those users.

   • Slack thread in #evss-prod: https://dsva.slack.com/archives/C8R3JS8BU/p1593635695339100
   • We need to make sure we have an account that works for the demo (like vets.gov.user+15@gmail.com)
   • Update: We contacted Raparthi, Mahesh (CSRA) <Mahesh.Raparthi@va.gov> from the EVSS Test Team to clear the EP codes for users 226, 228, and 229. We sent the users, email, and the SSNs for the sample users, to get the PIFs cleared. We asked what the threshold is for max EP codes and are awaiting a response.

3. We are testing form526 submission confirmation email in staging by submitting the form through the UI and are receiving an error that we did not see when testing in our review instance: uninitialized constant Form526ConfirmationEmailJob Did you mean? FormContactInformation.

   • There might be some configuration difference between environments?
   • Slack thread: https://dsva.slack.com/archives/CBU0KDSB1/p1593632394154600
   • Sentry log: http://sentry.vfs.va.gov/vets-gov/platform-api-staging/issues/139982/?query=is%3Aunresolved
   • Update: This seems to be job-specific, not user-specific. It may have to do with some staging configuration on the vets-api worker side regarding eager-loading.

Notes:

• we can ssh into the review instance, which is how we got the error message (get the ip address for ssh associated with the review instance from the jenkins log and run ssh dsva@<ip-...> )
• we needed to turn off ssoe for the review instance to be able to login with id.me (/flipper/features on the api url)
• list of users: https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/blob/master/Administrative/vagov-users/mvi-staging-users.csv

[ffafara-tw]

Shall we also try these users: https://github.com/department-of-veterans-affairs/va.gov-team-sensitive/blob/master/Administrative/vagov-users/staging-test-accounts-claims.md

[marisahoenig]

Created new tech task for getting dev environment to work: https://github.com/department-of-veterans-affairs/notification-api/issues/109