Architecture Design Intent - Privacy / Infrastructure - My VA forms status / confirmation page PDF storage and download

[tbaker1026]

Next Steps for the VFS team

• [ ] Assign this ticket to the team member(s) responsible for addressing feedback provided by Platform.
• [ ] Comment on this ticket:
  • If the Platform reviewer has any Thoughts/Questions that require responses.
  • When Must feedback has been incorporated. As appropriate, link to any other GitHub issues or PRs related to this feedback.
  • When Should/Consider feedback has been incorporated, or if any feedback will not be addressed. As appropriate, link to any other GitHub issues or PRs related to this feedback.
• [ ] Questions? For the most timely response, comment on Slack in your team channel tagging @platform-governance-team-members.
• [ ] Close the ticket when all feedback has been addressed.

Thoughts/questions

-

Feedback

Practice areas will document their feedback on the VFS-provided artifacts following the Must, Should, and Consider Framework. Platform Governance reviewers may also provide additional notes that don’t comment on the artifacts themselves but are important for implementation (eg. engineering/coding notes).

Governance team actions

• [ ] Format feedback as individual tasks (check boxes)
• [ ] Assign this ticket to the VFS team member that opened the Slack request
• [ ] Add the VFS team product label
• [ ] Add the VFS team the feature label (if applicable)
• [ ] Add the touchpoint labels
• [ ] Add the practice area labels
• [ ] Add the Collaboration Cycle initiative milestone

[tbaker1026]

@Thrillberg - Action items:  Provide average # of submissions and size of submissions of forms (lean towards the largest size)

[JonathanKamensDVA]

Thanks for meeting with the architecture intent team today. The consensus on our side of the aisle is that this was a very successful meeting, and we the project team felt the same way! We are very interested in any feedback you have about improving the process and making these meetings go more smoothly.

I am sorry that we forgot to make sure someone was taking notes contemporaneously during the meeting. The folks running this particular collab cycle stream are new to the process so it slipped our minds. We will make sure that is done moving forward!

Here are the actionable points and important takeaways I remember from the meeting. Hopefully others can add anything that I missed:

• We should put a ticket in the backlog about attempting to convince Lighthouse to allow our PDFs to be retained for longer, at which point we may be able to switch over to downloading PDFs directly from Lighthouse rather than storing them ourselves in S3. @JonathanKamensDVA has agreed to help with trying to convince Lighthouse to make the change. We will not hold up shipping this feature to wait for this change at Lighthouse; we will continue with the plan to store PDFs ourselves in S3.
• Rather than having PDFs stream through a vets-api endpoint to the user, we will investigate having vets-api generate signed S3 download URLs that can be sent down to the browser for the browser to download the PDF directly from S3. @JeffKeeneVAGov has agreed to point the team in the direction of where they can learn more about how to do this.
• We will look into ensuring that there is a Content-Disposition header on the S3 download with the correct filename in it so that the filename the user is prompted to save is correct. We think it is probably possible to achieve this by setting properties on the file when it is uploaded into S3.
• More thought needs to be put into ensuring that the PDFs are purged at the end of their desired save time. How this is done is an important part of the architecture, so we may want to have another review meeting to review the plan for this once we have one. This plan should include two things:
  • the purge mechanism itself; and
  • a process or mechanism for periodically auditing the contents of the bucket to ensure that the purge mechanism is working.
• We would like to see more details about what new events are going to be tracked to collect metrics related to this new functionality.
• We would like to see more details about the team's plan for monitoring this new functionality for performance, reliability, and actionable errors after it is rolled out.
• In addition to a mechanism for purging the PDFs at the end of the 60-day window, which also need a mechanism for purging other PII from the form_submissions table.
• We need to estimate how many forms we expect to be saved per unit of time and approximately how much space they will take up so we can determine how much this is going to add to our AWS bill. It probably won't be too expensive, but we need to make sure. We will need to repeat this exercise as we add additional forms in the future, so we aren't surprised with a big increase in our AWS bill.
• Consider uploading the PDF to S3 in a Sidekiq job rather than doing it synchronously. This will require frontend changes to display a message indicating that the PDF is still being uploaded which ideally changes automatically to the download link once the upload to S3 is completed. Doing the upload in a Sidekiq job will make the UI more responsive to the user and will provide for error-handling in cases when there is a short-term or long-term S3 outage when at the time a form is submitted.
• Steps should be taken to prevent the browser from pre-fetching the PDF when it sees the link on the page, since this will cost us money, impact site performance, and potentially throw off our metrics. This could be accomplished by making the link activate a JavaScript function that does the download, rather than an actual link that starts the download directly.
• We agreed that the 60-day clock on removing a PDF can be reset when the status of the form submission changes.
• For far future consideration, not to worry about right now: we are at some point going to need to decide whether the saved PDFs need to be mirrored to our disaster recovery AWS region, or whether it's OK for them to be lost in a disaster. The cost of storing the PDFs may influence this decision.
• We need to make sure that the permissions on the S3 bucket are locked down. @JeffKeeneVAGov has agreed to connect the team with the right people on the platform team to help make that happen.

[DonMcCaugheyUSDS]

Here are some helpful links related to the discussion:

• S3 Signed URLs
• Sidekiq Job Standards
• Endpoint monitoring
• Content-Disposition header

[DonMcCaugheyUSDS]

@Thrillberg and team -- thank you again for an excellent discussion. The feedback above is guidance based on our collective experience; please exercise your professional judgement throughout implementation as your understanding of the problem grows.

[Thrillberg]

@DonMcCaugheyUSDS @JonathanKamensDVA We've started work on the above issues that I've broken out for PDF storage. So far we've got the following summary of the findings on size requirements for storage:

Conclusion: For 11,050 submissions/month, the total cost (storage + PUT + GET) is $0.42/month. (<-- our current roughly) For 1 million submissions/month, the total cost is $39.90/month (storage + PUT + GET)

More details in the link above. Can you let me know if these costs seem prohibitive?

[DonMcCaugheyUSDS]

More details in the link above. Can you let me know if these costs seem prohibitive?

@Thrillberg This seems reasonable. I didn't expect this to be expensive but it's important to check intuition.

Checking one related assumption of mine: is a PDF only created when the form is submitted, or is it possible for a veteran to save a draft form, and if so, does that create a PDF? I want to double-check that 1 submission == 1 PDF stored in S3.

[Thrillberg]

@DonMcCaugheyUSDS A PDF is only created upon submission, so yes it will be 1 submission == 1 PDF stored. However! We're talking about possibilities where we might want to store 2 PDFs per submission, one with the signature and various watermarks and the other without. We'd want the one without to be available to the user to download and print so that if they mail it in, it doesn't falsely say that it was digitally submitted. For now, we're not opening that can of worms but it is a future possibility that we may want to elaborate on.

cc @humancompanion-usds

[DonMcCaugheyUSDS]

We're talking about possibilities where we might want to store 2 PDFs per submission, one with the signature and various watermarks and the other without.

Sounds good. Mainly wanted to check that assumption and I'm not worried about a 2x potential increase.