Analyze beta site analytics for opportunities

acolter commented 7 years ago

Dependent on https://github.com/18F/clinical-trials/issues/20

acolter commented 7 years ago

What platforms and browsers people are using
Pages with the highest read time
Search terms
Monthly / daily users
Most popular pages
Referral pages

austinhernandez commented 7 years ago

@MelissaBraxton this is what I was thinking in terms of an action plan for the analytics. It's a work in progress, but feel free to change this and add to it.

Additional information in #20

Mobile and desktop users:

Desktop users:

What this information tells me:

There's a whopping 3,043,220 users a users a week to the site.
2,600,329 a majority (85%) are coming from the desktop site, the rest from mobile
Most users come from Chrome on a Windows

Why does that matter?

Analytics won't tell you everything you need to know, its just a piece of a puzzle
But, with such a sizable pool of users so much cool stuff can be done to experiment and test
It's important for stakeholders to understand the traffic of the site so you can prioritize issues and requests
- "Is this an issue that effects 80% of users or 2%?
You can make assumptions on the users based on their browser and OS

How can I make this actionable:

Yes the site needs to work for everyone, but when launching small features at a time you want to start with the highest represented group.
You might want to look at finding more granular information like "What pages are they looking at?"

Goal setting

The most important thing about quantitative data is creating a hypothesis that can be measured
- "We believe if we do X then Y will happen"
An easy goal is "We want to reduce the amount of calls to our customer service team to ask for customer service number" or "We want to reduce the number of 'no results' search results!"
Next set a game plan, "We believe that if we add a contact section at the top of the form, we can achieve that." or "We believe the language is unclear on the search box about what you can search for."
Set up multi-variant (A/B tests) and see if there are any changes
A good resource on tracking the user journey - https://mixpanel.com/funnels/

austinhernandez commented 7 years ago

Note to self and @MelissaBraxton for working session Wednesday: A little birdie told me that @onezerojeremy did something similar to this for FEC.

MelissaBraxton commented 7 years ago

Awesome. And I am just now reading through your first take above. Great start! Thanks so much for getting the ball rolling. Looking forward to digging in a bit more on Wednesday!

Melissa Braxton, MS User Experience Designer, 18F PhD Candidate, Human Centered Design & Engineering University of Washington, College of Engineering Cell: (415) 696-4216 https://18f.gsa.gov/

On Mon, Mar 27, 2017 at 1:36 PM, Austin Hernandez notifications@github.com wrote:

Note to self and @MelissaBraxton https://github.com/MelissaBraxton for working session Wednesday: A little birdie told me that @onezerojeremy https://github.com/onezerojeremy did something similar to this for FEC.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/18F/clinical-trials/issues/21#issuecomment-289577957, or mute the thread https://github.com/notifications/unsubscribe-auth/ATscnk4DS_3Js_M3RiWF9z7oHf5EGCzjks5rqB24gaJpZM4MO9NM .

onezerojeremy commented 7 years ago

Oh cool! 15% mobile is pretty substantial!

Do we have access to the search logs?

austinhernandez commented 7 years ago

@onezerojeremy yes! the file they sent is here: CT.gov Feb 26 – Mar 4 analytics data

austinhernandez commented 7 years ago

Just looking at some of the search data:

137,516 total search results new learning from team: undefined or empty searches are probably API calls

102,690: undefined or empty searches- 75%
- (result returns: 240,252 studies found for: ALL)
23,552: searches for NCT########## - Approx 17%
1,400: terms that are long and specific e.g. "a twelve-week, double-blind, placebo-controlled, randomized, parallel-group, multicenter study of the safety and efficacy of jzp-110" or "efficacy, safety and tolerability of avp-786 (deuterated [d6]-dextromethorphan hydrobromide [d6-dm]/quinidine sulfate [q]) for the treatment of agitation in patients with dementia of the alzheimer"
270: 5 digit numbers (zip code?)
2,000: 6, 7, 8, or 9 digit number (what could these be?)

18F / clinical-trials