Evaluate website performance monitoring tools

[rianfowler]

Overview

• website performance is not measured or monitored on va.gov

Goals

• Evaluate third party performance measuring tools
• Determine process for measuring and reviewing impact of website performance (see related)

Potential tasks

• Review third party offerings
• Determine constraints for using SaaS products for code review
• Create a plan for running a few options through a trial
• Determine rollout plan

Code quality tools to evaluate

https://speedcurve.com/pricing/ https://www.datadoghq.com/rum/

[rianfowler]

Create overview

• product types (product agnostic)
  • RUM
  • Synthetic
• How do product types overlap with existing tooling like GA?
  • GA
  • prometheus (talk to keifer)
  • level of overlap
• High level of how each product type could be implemented as part of our CI
  • How would the build process change?
  • level of effort
  • Are there any third party scripts that need to be added to the site?
• Summary of offerings
  • type of product offered
  • first year, on going price (estimated according to site)
  • any pertinent notes (unique offering)
  • integration effort
  • approvals needed

Prototyping

• Identify / engage some stakeholders
  • apps teams PMs
  • apps teams FE
  • DEPO
  • platform PMs
  • platform FE
  • accessibility specialists
• pick 2 vendors and prototype integration / evaluate demo

Pick vendor

• get quotes
• get necessary approvals

[jhonnyoddball]

Google Analytics - Web Performance Monitoring

The only information related with web performance monitoring inside GA is located at: Reports -> Behavior -> Site Speed

Average loading times are important metrics.

[jhonnyoddball]

Prometheus - Web Performance Monitoring

it does not provide any real and liable data to be used for performance metrics

[jhonnyoddball]

Web Performance Monitoring Tools

Synthetic Monitoring - The site receives a check once a minute or so RUM - Receives the data directly from the site because the script runs on the page.

Monitoring Tool	RUM	Synthetic	Price	*Score
Akamai - mPulse	X		?	1
DotCom-Monitor	X	X	Confusing	1
Uptrends	X	X	Varies, $50 p/m + RUM usage	2
DareBoost		X	$356 p/m	3
RayGun	X		$649 p/m	1
Rigor		X	?	2
DataDog	X	X	?	4
SpeedCurve	X	X	$1,500 p/m	5

• Score is based on amount of services offered, where 1 is the most simple provider and 5 is the one that offers the most tools and metrics.

[jhonnyoddball]

AKAMAI - MPULSE

https://www.akamai.com/us/en/products/performance/mpulse-real-user-monitoring.jsp https://developer.akamai.com/akamai-mpulse-real-user-monitoring-solution

• Mainly a RUM solution
• Most information can be found in GA
• Act on Problematic Resources
• Provide alerts

PRICE: Upon request DEMO: Upon request. SCORE: 1

[jhonnyoddball]

DOTCOM-MONITOR

https://www.dotcom-monitor.com/website-speed-monitoring/

• Mostly RUM with a little of Synthetic
• Detailed waterfall charts show root cause of issues.
• Video playback and screenshots of errors.
• Historical & real-time element reporting.
• Provide alerts

PRICE: Confusing (https://www.dotcom-monitor.com/pricing/) DEMO: Upon request. SCORE: 1

[jhonnyoddball]

UPTRENDS

https://www.uptrends.com/products/web-performance-monitoring

• Includes RUM and Synthetic
• Get alerted on website speed, the total number of bytes, or missing content
• Third party monitoring
• Provide alerts

PRICE: Varies, $50 p/m + RUM usage (https://www.uptrends.com/pricing) DEMO: Upon request. SCORE: 2

[jhonnyoddball]

DAREBOOST

https://www.dareboost.com/en

• Website Speed Test
• Installation-free (only Synthetic)
• HTML, images, JavaScript, CSS analysis
• Tool provides solutions, ordered by priority
• Comparison feature
• Performance budget tool
• Custom alerts
• User Journey - Allows you to replicate any user navigation scenario

PRICE: $356 p/m (https://www.dareboost.com/en/offers#ifbefs_nfov) DEMO: Upon request. Free Report (https://www.dareboost.com/en/report/a_15df7939f90b9981b244f29ce) SCORE: 3

[jhonnyoddball]

RAYGUN

https://raygun.com/

• Real User Monitoring in minutes using a lightweight library
• Automatically identify front-end performance issues causing slow and frustrating load times.
• Mobile app performance

PRICE: $649 p/m (https://raygun.com/platform/real-user-monitoring#rum-business) DEMO: Upon request. SCORE: 1

[jhonnyoddball]

RIGOR

https://rigor.com/

• Synthetic monitoring with an intelligent optimization engine
• Integrate Performance Management Throughout the DevOps Lifecycle
• Seamless Integration with Your Favorite Real User Monitoring (RUM) Solution
• Test Performance in Your Jenkins Build with the New Rigor Optimization Plugin

PRICE: No pricing. Only after trial. DEMO: Upon request. SCORE: 2

[jhonnyoddball]

DATADOG

https://www.datadoghq.com

A lot of backend integration.

RUM (https://www.datadoghq.com/rum/)

• Visualize load times, frontend errors, and page dependencies
• Correlate business and application metrics
• Troubleshoot quickly with app, infrastructure, and business metrics in a single dashboard
• Uncover an incident's root cause with full-stack context

Synthetic (https://www.datadoghq.com/synthetics/)

• Validate HTTP requests to your services from several locations
• Out-of-the-box integration with APM, infrastructure, and logs
• Waterfall visualization of each step with screenshots of end-user view
• Breakdown of performance by selected locations and devices
• Fast root cause analysis
• Record user actions via a GUI-based Web Recorder
• Powerful validations with use of flexible assertions and variables
• Empower non-technical teams to create browser tests
• See metrics, traces, and logs in the context of test runs
• Visualize uptime data alongside key business metrics
• Latency display of DNS, SSL, TTFB, and more.
• Real-time alerting with rich context

Integration, dashboard, alerts, log management, network, monitoring, and API are features but sold separately.

PRICE: Varies based on product bundle (https://www.datadoghq.com/pricing/) DEMO: Upon request. SCORE: 4

[jhonnyoddball]

SPEEDCURVE

It is by far the most complete of all. It has both RUM and synthetic monitoring. It has most of all competitor's features and more.

• Front-end performance monitoring
• Track asset sizes and requests
• Built in Google PagesSpeed Insight reports
• Weekly reports and alerts
• Built on top of WebPagetest
• Responsive website testing
• Competitive benchmarking
• Share dashboards with the team
• API to trigger tests on every deploy
• Visual diffs to spot UX regressions

RUM (https://speedcurve.com/features/lux/)

• A lot of metrics
• Page Details
• Navigation timing
• Paint timing
• CPU time
• User timing
• Page design
• Interaction Metrics
• Customer Data
• Javascript Errors

Data collected by RUM: https://support.speedcurve.com/en/articles/1707704-what-data-is-collected-by-lux-for-rum

Metric representation: https://support.speedcurve.com/en/articles/74078-what-do-the-different-speedcurve-metrics-represent

Synthetics (https://speedcurve.com/features/synthetic/)

monitoring is built on top of WebPageTest,

• Track dozens of metrics, including custom metrics
• Create performance budgets and get alerts
• Diagnose performance issues at the page level (Provided with Google's Lighthouse & PageSpeed Insights audits, there is a list of performance rules, stacked and color-coded in their order of importance.)
• Analyze the performance impact of ads and third parties

SpeedCurve CLI with Node (https://github.com/SpeedCurve-Metrics/speedcurve-cli)

• Triggering deploys
• Monitoring the status of deploys
• Getting the status of performance budgets
• Fetching synthetic test data
• Listing sites in an account
• JSON output for easy integration into build pipelines

PRICE: $18,000 per year for RUM and Synthetic (https://speedcurve.com/pricing/) DEMO: Demo available SCORE: 5

[jhonnyoddball]

SpeedCurve - Performance Budget

More Info on how to create a performance budget

Allows you to set a specific metric budget only per chart. there are about 50 different metrics

Settings:

Chart:

[jhonnyoddball]

SpeedCurve - Synthetic

Highlights

Site

Provides you with the most import metrics

• Filmstrip
• Start Render
• SpeedIndex
• Backend (TTFB)
• Page Load
• Fully Loaded
• All LightHouse scores (Performance, Progressive Web App, Accessibility, Best practices and SEO)
• Time to Interactive
• Blocking JS
• Blocking CSS
• Content requests
• Content Sizes

Deploy

Allows you to compare the same site into 2 different instances. Just run the test on the days you want to perform and compare. It could be helpful to compare the progress of improvements.

Responsive

Shows you the responsiveness of among tested devices

Assets

Summary of the requests

• Slowest requests
• Largest requests

Third Party

Recognizes and displays all third-party information

Javascript

Displays all information and metrics related with js files

Benchmark

Displays metrics from all the competitors

Synthetic tab can't be modified, however, any other metrics/custom metrics can be added to any dashboard under Favorites

[jhonnyoddball]

SpeedCurve - Improve

It displays the audits and recommendations from LightHouse

[jhonnyoddball]

PERFORMANCE METRICS

Recommended metrics from USWDS

This should be the main metrics you should focus on for Web Performance. SpeedCurve has all of them.

• Speed Index: it’s the average time at which visible parts of the page are displayed. It can only be used in a simulated environment
  • WebPageTest
  • SpeedCurve
• First Meaningful Paint: It’s the biggest above-the-fold layout change has happened, and web fonts have loaded
  • LightHouse
  • Chrome DevTools
  • SpeedCurve
• First Contentful Paint: It's the time at which users can start consuming page content.
  • SpeedCurve
• Time to Interactive: It measures how long it takes a page to become interactive
  • LightHouse
  • Chrome DevTools
  • SpeedCurve
• Input Latency: Only available in RUM. It’s the amount of time it takes for the app to respond to the users as they interact with it.
  • LightHouse
• Start Render: It’s measured as the time from the start of the initial navigation until the first non-white content is painted to the browser display
  • WebPageTest
  • SpeedCurve
• Backend Time or TTFB: The time from the start of the initial navigation until the first byte of the base HTML page is received by the browser
  • SpeedCurve
• Number of requests
  • LightHouse
  • SpeedCurve
• Total Size
  • LightHouse
  • SpeedCurve
• Blocking CSS
  • LightHouse
  • SpeedCurve
• Blocking JS
  • LightHouse
  • SpeedCurve
• LightHouse scores
  • LightHouse
  • SpeedCurve

Most of the above metrics can be found in SpeedCurve under Synthetic > Site. However, FCP and FMP can only be found in a custom Dashboard or in the cascade chart.

SpeedCurve reply:

“You can find First Contentful Paint and First Meaningful Paint when you drill down into your Synthetic test results. You can also create custom charts that track FCP (in LUX and Synthetic) and FMP (Synthetic only) in your Favorites dashboards, as well as create performance budgets and alerts for them.”

[jhonnyoddball]

SpeedCurve - Metrics test

Parameters

Comparison Test

Metric	VA	HealthCare	HealthData
Speed Index	2.43 s	3.88 s	0.83 s
First Meaningful Paint	1.92 s	2.05 s	1.05 s
First Contentful Paint	1.92 s	1.47 s	0.76 s
Time to Interactive	2.71 s	5.27	0.76 s
Start Render	1.8 s	1.5 s	0.8 s
TTFB	0.7 s	0.35 s	0.2 s
Number of requests	43	78	37
Total Size	923 KB	2,594 KB	330 KB
Blocking CSS	2	4	9
Blocking JS	2	11	6

LightHouse Scores

Metric	VA	HealthCare	HealthData
Performance	51	28	94
PWA	58	31	54
Accessibility	100	92	97
Best Practice	86	86	86
SEO	100	80	73

[brandonrapp]

What would do we think will be useful for VSP and VFS teams? What decisions do you think we can make using the data from speedcurve? How could VSP use this to identify optimizations and test out assumptions?

[jhonnyoddball]

SpeedCurve - Github authentication

Their response:

Hi there! We don't offer GitHub authentication at the moment, but we do support other forms of SSO including SAML, ADFS, and Google G Suite.

[jhonnyoddball]

SpeedCurve - Usefulness

What would do we think will be useful for VSP and VFS teams?

• All metrics available
• Dashboards per team(s)
• Performance budget
• Snapshots in time (deploy)
• Custom metrics
• Slack integration

What decisions do you think we can make using the data from speedcurve?

Definitely a lot. Starting with the Improve tab. We will be able to check what need to be done in every page been tested.

How could VSP use this to identify optimizations and test out assumptions?

There are several metrics that can be used to improve the overall performance in each site component. We can always separate each instance and separate it among team or at least a bigger group.

Snapshots can also be useful to identify the progress made and what can be affecting the site as time passes by.