Assess cost of implementing Datadog Real User Monitoring (RUM) on Benefit hub / detail pages

jilladams commented 8 months ago

User Story or Problem Statement

As a platform owner, I need to understand cost implications of using Datadog RUM on additional pages, to manage materials costs.

Description or Additional Context

department-of-veterans-affairs/va.gov-cms#15636 defines implementing Datadog RUM on the Health care Benefit hub and its child Benefit Detail Pages.

Adrian Rollett flagged that RUM has cost implications.

We need to:

Develop an idea of how many pageviews will be involved for this set of pages. (anything under /health-care/)
Do a order of magnitude cost estimate, based on Datadog's pricing info. (We don't know if VA receives any discount.)
Review with Ray Wang and Danielle Thierry for approval.

Acceptance Criteria

[ ] Cost estimate is assembled
[ ] Estimate is provided to Dave Conlon, @raywangoctova and @DanielleThierryUSDSVA for approval
[ ] department-of-veterans-affairs/va.gov-cms#15636 is either unblocked or closed as no-op, depending on outcome of the review

jilladams commented 8 months ago

@FranECross @wesrowe Confirmed we need to assemble this information, which will require Analytics access. This is not necessarily urgent, BUT: it's unclear when Fran will get Analytics access, so it might be nice to try and figure this out before Wes moves off the team, via his access. Please discuss and let me know if that's not reasonable / doable, and we can figure it out with Dave if that's not realistic.

jilladams commented 8 months ago

Ray Messina says: Platform doesn't handle Datdog billing. That is a DOTS thing.

Reached out to #dots-dsva / Tinu, for information on 10/24: https://dsva.slack.com/archives/C01U8EYHSEM/p1698165407788909

No response yet. Pinged again today.

jilladams commented 8 months ago

From: https://dsva.slack.com/archives/C01U8EYHSEM/p1698783407837599?thread_ts=1698165407.788909&cid=C01U8EYHSEM and down in the thread

Pilot: Dave confirmed: we do want to go ahead with cost analysis for this "pilot"
- Pilot should be focused on /healthcare/ benefit hub and sub pages
- Goal is to set some % of traffic that we can monitor, that will display trends of page use as unauthed users move into the authed healthcare experience
Can see current RUM applications here: https://vagov.ddog-gov.com/rum/list
- Most seem to be configured at 100% of user sessions, with 20% for session replay.
- This works out to around 500k user sessions per month per app, which is about $780/mo with default pricing before bulk discounts.
Cost analysis: We need to
- figure out: the approximate traffic volume of the page(s) where we want to use RUM.
- Reference this example of cost analysis that Steve Albers provided to Bill Chapman and Mike Chelen for a RUM implementation his team did for 526EZ: https://github.com/department-of-veterans-affairs/va.gov-team/issues/61680
- Assemble our analysis and pitch back to (?) for review - tbd if reviewers are Bill Chapman / Mike Chelen for this as well. I asked here: https://dsva.slack.com/archives/C01U8EYHSEM/p1698953736989299?thread_ts=1698165407.788909&cid=C01U8EYHSEM

jilladams commented 8 months ago

@mnorthuis we discussed this thread in #dots-dsva with @davidconlon today and he requested that we ask you to help put together the % and traffic data proposal, that he can review with his Platform hat on for approval, and collab to get that to DOTS. We can move this ticket back over into your repo if you'd like?

jilladams commented 8 months ago

FYI to @coforma-terry that I transferred this issue back to va.gov-team repo and added the sitewide-CAIA label, based on notes in comments above.

coforma-terry commented 8 months ago

Thank you for the heads-up @jilladams

mnorthuis commented 6 months ago

@jilladams Cost estimate for RUM implementation within the health care benefits hub

User sessions within the unauthenticated www.va.gov/health-care/ space

Data omits sessions for check-in, pre-check which already has RUM implemented
Data omits sessions for 10-10EZ application, which already has RUM implemented
Data omits sessions for appointments, which is an auth experience
Note: there are 2 efforts that will eventually reduce the pages and sessions within this section - the family member hub work, and streamlining of benefit hubs
Data pulled from a 12 week period of 8/6/23-10/28/23, includes a spike of sessions across 2 days due to PACT Act that were 3-5 times higher than normal

Avg user sessions per week: 128,144 $2.25/1000 sessions (includes GovCloud in rate) ~$300/week

In comparison to other apps currently running RUM - 12/3/23 - 12/9/23: Check-in - 168k sessions/week MHV on VA.gov - 150k sessions/week 10-10EZ - 30k sessions/week

Track small percentage of traffic for purposes of pilot 10% of traffic, 6 mo pilot

jilladams commented 6 months ago

@raywangoctova @DanielleThierryUSDSVA @davidconlon Mikki has provided RUM cost estimate for implementing on Benefit Hubs in the comment above, for your review / approval.

raywangoctova commented 4 months ago

Hi @jilladams - Someone else should be providing the approval since I need to manage the Data Dog purchase.

jilladams commented 4 months ago

@DanielleThierryUSDSVA @davidconlon is one of you able to approve Mikki's proposal?

DanielleThierryUSDSVA commented 4 months ago

Hi, @jilladams. Just confirming that I'm not the approver on this. @davidconlon will follow up on this when he has more info to share. Thanks!

DanielleThierryUSDSVA commented 3 months ago

Bringing @jstrothman @va-albers in here, who are looking at DataDog on some unauth pages.

mnorthuis commented 1 month ago

@davidconlon Can we get an update on this? With the family hub launching soon, its another place where this data would be extremely helpful. What are the barriers or next steps? Or please let me know if someone else should take this over.

department-of-veterans-affairs / va.gov-team