Groom analytics for Auth Exp KPI dashboard

joanneesteban commented 4 years ago

Issue Description

Groom analytics for technical feasibility.

Tasks

[x] Document data sources for each metric
[x] Document unavailable vs. available metrics
[x] If any, add implementation issue for additional necessary implementation

Acceptance Criteria

[x] We understand what can be added to the KPI dashboard, what will need more sourcing, and what it takes to get there

bmcgrady-ep commented 4 years ago

Auth Experience KPI	Metric	Source	Go/No Go
Overall searches for "direct deposit"	Total Unique Searches of "direct deposit"	Google Analytics	✔️ Go
Searches for "direct deposit" initiated from the profile	Total Unique Searches of "direct deposit"	Google Analytics	✔️ Go
Searches initiated from the profile	Total Unique Searches	Google Analytics	✔️ Go
Pages visited before the profile	Previous page before va.gov/profile/index.html	Google Analytics	✔️ Go
Time spent on profile	Average time on page for profile	Google Analytics	✔️ Go
ForeSee score rating	ForeSee score data	ForeSee	Future 🔶 Go

joanneesteban commented 4 years ago

Final Touches!

@amycesal @bmcgrady-ep

Not much left, looks really good so far!

[x] Asked Matt if he has rollout goals. We should add these goal lines if they do.
[x] "OKR"s should be "KR"s
[x] Pages visited before the profile: it looks like we need to work through this visualization. It also looks like the previous page path isn't concatenating the data? Also, what's the date range for this?
[x] Time spent on profile pages: it ooks like we need to work through this visualization as well. Is it possible to add their goal?
[x] ForeSee data needs to get switched to include /profile instead of all. Should this be a bar chart or a line graph?
[x] Add data source links where possible

Stretches

[ ] Overall searches for "direct deposit": working with the team to find out the baseline so we can add a goal line for 20% reduction

joanneesteban commented 4 years ago

^ForeSee data won't have anything until after they launch

joanneesteban commented 4 years ago

Added Quick FIlters to the ForeSee chart so they can distinguish between the different URLs.

bmcgrady-ep commented 4 years ago

@joanneesteban - Will they have ForeSee data once they launch to 25% or 100%?

cc: @mshea0606

jonwehausen commented 4 years ago

QA of Auth Experience Dashboard - BigQuery

Description of QA Effort	Screenshot	Status & Notes
Verified Overall searches for "direct deposit" card is configured for explictly the strings "direct deposit" or direct deposit		:heavy_check_mark PASS Total searches is not a metric that directly lives within GA so doesn't have an exact report equivalent, that said -- the portion of the query defining total and unique searches look in accordance with best practices this could be expanded to include anywhere where direct deposit is said in the whole search query simply by changing this filter at the card level to CONTAINS
Verified Searches for "direct deposit" initiated from the profile card is configured to include each of the relevant `/profile` pages and for "direct deposit" search keyword		This dataset is the same dataset as the one above (leverages the same select statement lines for total and unique searches) Total searches initiated from profile is a metric we are creating via SQL that doesn't have an exact GA equivalent. That said filtering on `previous_page_path` of `/profile` as in the screenshot will get us a close estimate of searches initiated from `profile
Verified Searches initiated from the profile is configured to filter for `previous_page_path` of all relevant `/profile` pages		This dataset is the same dataset as the other two above (leverages the same select statement lines for total and unique searches) Again, total searches initiated from anywhere profile is a metric we are creating via SQL that doesn't have an exact GA equivalent. That said filtering on `previous_page_path` of `/profile` as in the screenshot will get us a close estimate of total searches initiated from `profile
Verified Pages visited before the profile is configured to filter on a `page_path` of direct deposit pages (i.e `www.va.gov/profile/direct-deposit`) and a `previous_page_path` of `/account`		:heavy_check_mark PASS This dataset specifically grabs previous page paths relating to profile. Numbers were tested against 9/7 matching GA precisely.
Verified Average time spent on profile is accurately depicting average time on a weekly basis (no card filtering needed)		:heavy_check_mark PASS This calculation will not match GA exactly as BigQuery produces a more precise average time on page based on the average of all page paths. In this example, we can see the number in the chart is a representation of the true average (average of all of the average time on page associated with profile) This was tested by tallying up the averages of each of the profile pages and manually performing the average calculation (as BigQuery is doing)

joanneesteban commented 4 years ago

Feedback

joanneesteban commented 4 years ago

@bmcgrady-ep were you able to work through the spike issue in your call with Matt & Samara? They were looking to get an average without accounting for the spike. (One of the workarounds is to look at the Median instead of the Average of the dataset.)

Let me know what the update was there, and we can close this issue if that was addressed.

bmcgrady-ep commented 4 years ago

@joanneesteban - It doesn't look like DOMO allows us to use Median instead of Average, due to this being a Beast Mode Calculation. However, I found a way to smooth out the outliers and insert the average. Above what number would you consider to be an outlier? I currently have 300.

joanneesteban commented 4 years ago

@bmcgrady-ep right now, 255.92 is the outlier. Is there a way for them to easily export the data and do the calculations outside Domo? Otherwise, what would smoothing out the outliers and inserting the avg look like? If we remove the outliers, we need to explicitly call that out and ask if that's what they want for a long term dashboard since we're also looking to see spikes and dips.

bmcgrady-ep commented 4 years ago

@mshea0606 - would you rather have the spikes in Average Time on Profile or would you rather it be smoothed out to average?

bmcgrady-ep commented 4 years ago

@joanneesteban In this conversation with Matt, he said that spikes are okay to include in Average Time on Profile.

mshea0606 commented 4 years ago

hey @bmcgrady-ep, per our discussion, could you please change the Foresee score visual from quarterly to weekly on the Profile 2.0 Domo dashboard?

Also, could you add a total interactions by page metric/visual to the dashboard? (by week) Example: Count of Personal and Contact Information updates Count of Direct Deposit updates Count of Account Security updates Count of Connected Apps updates

Related to key result: Increased profile engagement/interactions

Thanks for your help with this!

cc/@Samara-Strauss @joanneesteban @jonwehausen @bsmartin-ep

Samara-Strauss commented 4 years ago

☝️ I think we can close this issue once Matt's questions above are addressed.

bmcgrady-ep commented 4 years ago

@mshea0606 - Made the updates to the Profile 2.0 dashboard as requested. For the total interactions by page metric section, each one of the charts has the total interactions for each page. I can change that to only include updates for each page but wanted to get your input.

Samara-Strauss commented 4 years ago

I know I said we could prob close this out, but I have another question — is it possible to set up a segment of people who have access to the Direct Deposit feature in the profile (so they can access the feature but don't necessarily have to have visited the page in a session)? We show this section conditionally based on whether you are eligible for direct deposit payments or not, so not everyone sees it.

Why I'm asking: I was pretty shocked to see that searches for "direct deposit" from the profile have actually not gone down since launch. In fact, they've gone up some weeks! This doesn't make any sense — direct deposit is clearly in the side navigation if you have access to the feature. There's no way people with access to the feature are missing it.

So, that led me to a new hypothesis: maybe the issue was not that we weren't clearly elevating direct deposit in the old profile, but that there were people who thought they should see this feature that were not seeing it. I'm trying to figure out a way to prove out this hypothesis. By looking at popular search terms, I can't tell whether the people initiating those searches can actually see direct deposit or not (I'm guessing no), but if I can determine that the reason people are searching is because they are not seeing that feature, then I've got something to move forward with. Let me know if you have any ideas on if we can set up a segment of some sort — whether it's people who have access to direct deposit (not sure that's possible) or something else. People who visit direct deposit actually may be an OK proxy.

Samara-Strauss commented 4 years ago

And sorry, the above request is for a segment in GA — no need to update Domo now, though this might lead to a Domo update.

Samara-Strauss commented 4 years ago

☝️ giving this a gentle nudge. Any thoughts?

bmcgrady-ep commented 4 years ago

Hi @Samara-Strauss - thanks for the nudge! Looking into this, I want to make sure I am understanding this correctly and have a quick question. Do all authenticated users have access to the Direct Deposit part of the profile? I assume this is true, but want to make sure.

Samara-Strauss commented 4 years ago

Do all authenticated users have access to the Direct Deposit part of the profile?

No, they do not.

All authenticated users have access to the profile.
If you are LOA1, you only have access to the Account Security page.
If you are LOA3, you have access to Personal Information, Military Information, Account Security, and Connected Apps by default.
If you are LOA3 and receive disability compensation & pension payments from the VA, you see the Direct Deposit section. If you do not receive disability compensation & pension payments from the VA, you do not see the Direct Deposit section even if you are LOA3.

Let me know if you have questions about that.

Samara-Strauss commented 4 years ago

So, basically I need a segment either of people who can see the direct deposit section, or if that's not possible, we can use people who update direct deposit and people who do not update direct deposit as proxies.

bmcgrady-ep commented 4 years ago

@Samara-Strauss - Here is a report showing "direct deposit" searches on profile pages. There are two custom segements applied - 1 for authenticated users who updated direct deposit in the session and 1 for authenticated users who did not update direct deposit in the session.

Samara-Strauss commented 4 years ago

Hey @bmcgrady-ep, am I missing something? I didn't see any segments in the report when I clicked that link:

bmcgrady-ep commented 4 years ago

@Samara-Strauss - I believe I need to share the segments with you:

Samara-Strauss commented 4 years ago

Should I select the WBC view or All data view, or does it not matter?

bmcgrady-ep commented 4 years ago

Use the All Data view

Samara-Strauss commented 3 years ago

I have a question: I checked out the results for the direct deposit segments mentioned above on searches made from /profile/personal-information (the first page of the profile) and was surprised to see that people who do update direct deposit and people who do not update direct deposit somehow don't make up 100% of searches from the profile. This is puzzling to me, because people either do or do not take this action from the profile; there's no middle ground. However, a third group — "all users" — rounds out the search counts (screenshots below) for searches made from /profile/personal-information.

I'm wondering if there is something wrong with the segments. For users who visit the profile, you either do or do not update direct deposit, and everyone should fit into one of those two categories. Why are we seeing "all users" round out the search count from the profile? What is that capturing that the direct deposit segments are not?

Let me know if you have questions. I can hop on a quick call if this is unclear.

bmcgrady-ep commented 3 years ago

@Samara-Strauss - I had the two segments set at session-level and have now switched them over to user-level segments. This more accurately segments the users into two distinct groups - 1 group who has updated their direct deposit and 1 group who has not updated their direct deposit. Here is the GA report with the two segments. Let me know if this helps!

bmcgrady-ep commented 3 years ago

Per discussion with @mshea0606, here are a few updates to the Profile 2.0 KPI dashboard to be made:

Add Total Submissions to the "Increase Profile Engagement/Interactions" section
If it is possible add weekly count of survey submission card for Foresee data.

mshea0606 commented 3 years ago

@bmcgrady-ep Would it be possible to also add "Total profile submissions" by week to help determine profile engagement?

bmcgrady-ep commented 3 years ago

@mshea0606 - Thanks for taking the time to meet with me. I added the Total Interactions on Profile chart by week and the Total Profile Submissions.

Samara-Strauss commented 3 years ago

What is "Total profile submissions"? I have assumptions, but want to make sure I'm clear.

bmcgrady-ep commented 3 years ago

What is "Total profile submissions"? I have assumptions, but want to make sure I'm clear.

Total Profile Submissions are any profile saves. I can change the name if you and @mshea0606 have a different name you would like to use.

Samara-Strauss commented 3 years ago

Thanks! That's what I assumed. No need for a name change.

bmcgrady-ep commented 3 years ago

@mshea0606 - I updated the KPI dashboard to show the number of ForeSee survey submissions by week. Let me know if there is anything else you need!

Samara-Strauss commented 3 years ago

I'm going to close this. You all have been wonderful. We can open a new ticket if we have any additional requests.

department-of-veterans-affairs / va.gov-team