Events v2 optimization - content build slowdown due to code changes

brianalloyd commented 2 years ago

Description

Around Jan 3 2022, the average content build time became approximately 5 minutes slower: https://app.datadoghq.com/dashboard/eb9-ece-f9z/platform-cms-prod-cms?fullscreen_end_ts=1642813745148&fullscreen_paused=false&fullscreen_section=overview&fullscreen_start_ts=1640135345148&fullscreen_widget=7727346438800838&from_ts=1640369100921&to_ts=1643047500921&live=true

There does appear to be some change in GraphQL time at this point, but it does not explain the shift. Taking GraphQL time out of total build time shows that the non-GraphQL time has also jumped: https://app.datadoghq.com/dashboard/gm3-xnt-et5?fullscreen_end_ts=1643047588779&fullscreen_paused=false&fullscreen_section=overview&fullscreen_start_ts=1640369188779&fullscreen_widget=8055303753625958&from_ts=1640369184121&to_ts=1643047584121&live=true

The jump almost certainly happened as we came out of code freeze. Therefore, there is a range of work that could be responsible for the change. This is the full range of commits from 12/23 - 1/5, beginning of code freeze until just beyond: https://github.com/department-of-veterans-affairs/content-build/compare/22763456b64f001903126c2cebe2a38b0fe8c561...4b29aa4bb7405a01fb25255811d545da9fe2d934

We should do timeboxed investigation into this to identify the changes that have caused the slowdown. If fixing the problem is more involved than an hour's work, a follow up ticket should be created with the team which owns the code in question to rectify the problem.

Considerations

This is not an immediate concern, because we have mitigated the worst impacts of the issue by archiving any events older than 6 months. However, if the number of Events increases, or product needs dictate that we show historical events older than 6 months, the problem may recur. In particular, this is not an issue for Events v2 launch, which is imminent.
We'd recommend that this be done early/mid Q2, as works for Public Websites and the contract change that is happening. Platform CMS will continue to monitor content release and if Events causes issues with content release, we will escalate to the Public Websites team.
More detailed technical information on the problem can be found here: https://github.com/department-of-veterans-affairs/va.gov-cms/issues/8011#issuecomment-1055860857

Frontend Tasks

[ ] JSON data that Events v2 needs to function on event listing pages is currently being written directly to the HTML for those pages. This is duplicative and causes content release slowdowns when there are many events.
[ ] The recommended approach would be to externalize the JSON data needed for Events v2 to a single file that can be referenced by the client side as needed. Platform CMS can work with Public Websites on approaches to create this file as part of content build.
[ ] Public Websites would need to make any modifications to graphQL request that generates the JSON data, and any modifications to the front-end code, to ensure that Events continues to function as it does now while using the single JSON data file.

Acceptance Criteria

[ ] Events V2 is optimized to help speed content build forward.
[ ] Events is tested and working as intended on production environment
[ ] Follow-up work is planned with the team that owns the changes, to fix the issue

jilladams commented 2 years ago

@ethanteague and @maxx1128 this is an old ticket but I'm curious if this came up/ was addressed during your recent work. (cc @timcosgrove in case you have any recent notes since #8801 )

ethanteague commented 2 years ago

@ethanteague and @maxx1128 this is an old ticket but I'm curious if this came up/ was addressed during your recent work. (cc @timcosgrove in case you have any recent notes since #8801 )

No, it sure didn't / wasn't on our radar, @jilladams. That said, we now have a pattern we can follow = KISS architecture based on Covid status endpoint file that is accessible without SOCKS.

jilladams commented 2 years ago

Closing: https://dsva.slack.com/archives/C52CL1PKQ/p1659566857747289?thread_ts=1659545826.901579&cid=C52CL1PKQ

department-of-veterans-affairs / va.gov-cms