Closed joanneesteban closed 3 years ago
Auth Experience KPI | Metric | Source | Go/No Go |
---|---|---|---|
Overall searches for "direct deposit" | Total Unique Searches of "direct deposit" | Google Analytics | ✔️ Go |
Searches for "direct deposit" initiated from the profile | Total Unique Searches of "direct deposit" | Google Analytics | ✔️ Go |
Searches initiated from the profile | Total Unique Searches | Google Analytics | ✔️ Go |
Pages visited before the profile | Previous page before va.gov/profile/index.html | Google Analytics | ✔️ Go |
Time spent on profile | Average time on page for profile | Google Analytics | ✔️ Go |
ForeSee score rating | ForeSee score data | ForeSee | Future 🔶 Go |
@amycesal @bmcgrady-ep
Not much left, looks really good so far!
previous page path
isn't concatenating the data? Also, what's the date range for this?/profile
instead of all. Should this be a bar chart or a line graph?Stretches
^ForeSee data won't have anything until after they launch
Added Quick FIlters to the ForeSee chart so they can distinguish between the different URLs.
@joanneesteban - Will they have ForeSee data once they launch to 25% or 100%?
cc: @mshea0606
Description of QA Effort | Screenshot | Status & Notes |
---|---|---|
Verified Overall searches for "direct deposit" card is configured for explictly the strings "direct deposit" or direct deposit | |
:heavy_check_mark PASS Total searches is not a metric that directly lives within GA so doesn't have an exact report equivalent, that said -- the portion of the query defining total and unique searches look in accordance with best practices this could be expanded to include anywhere where direct deposit is said in the whole search query simply by changing this filter at the card level to CONTAINS |
Verified Searches for "direct deposit" initiated from the profile card is configured to include each of the relevant /profile pages and for "direct deposit" search keyword |
|
This dataset is the same dataset as the one above (leverages the same select statement lines for total and unique searches) Total searches initiated from profile is a metric we are creating via SQL that doesn't have an exact GA equivalent. That said filtering on previous_page_path of /profile as in the screenshot will get us a close estimate of searches initiated from `profile |
Verified Searches initiated from the profile is configured to filter for previous_page_path of all relevant /profile pages |
This dataset is the same dataset as the other two above (leverages the same select statement lines for total and unique searches) Again, total searches initiated from anywhere profile is a metric we are creating via SQL that doesn't have an exact GA equivalent. That said filtering on previous_page_path of /profile as in the screenshot will get us a close estimate of total searches initiated from `profile |
|
Verified Pages visited before the profile is configured to filter on a page_path of direct deposit pages (i.e www.va.gov/profile/direct-deposit ) and a previous_page_path of /account |
|
:heavy_check_mark PASS This dataset specifically grabs previous page paths relating to profile. Numbers were tested against 9/7 matching GA precisely. |
Verified Average time spent on profile is accurately depicting average time on a weekly basis (no card filtering needed) | :heavy_check_mark PASS This calculation will not match GA exactly as BigQuery produces a more precise average time on page based on the average of all page paths. In this example, we can see the number in the chart is a representation of the true average (average of all of the average time on page associated with profile) This was tested by tallying up the averages of each of the profile pages and manually performing the average calculation (as BigQuery is doing) |
Feedback
@bmcgrady-ep were you able to work through the spike issue in your call with Matt & Samara? They were looking to get an average without accounting for the spike. (One of the workarounds is to look at the Median instead of the Average of the dataset.)
Let me know what the update was there, and we can close this issue if that was addressed.
@joanneesteban - It doesn't look like DOMO allows us to use Median instead of Average, due to this being a Beast Mode Calculation. However, I found a way to smooth out the outliers and insert the average. Above what number would you consider to be an outlier? I currently have 300.
@bmcgrady-ep right now, 255.92 is the outlier. Is there a way for them to easily export the data and do the calculations outside Domo? Otherwise, what would smoothing out the outliers and inserting the avg look like? If we remove the outliers, we need to explicitly call that out and ask if that's what they want for a long term dashboard since we're also looking to see spikes and dips.
@mshea0606 - would you rather have the spikes in Average Time on Profile
or would you rather it be smoothed out to average?
@joanneesteban In this conversation with Matt, he said that spikes are okay to include in Average Time on Profile
.
hey @bmcgrady-ep, per our discussion, could you please change the Foresee score visual from quarterly to weekly on the Profile 2.0 Domo dashboard?
Also, could you add a total interactions by page metric/visual to the dashboard? (by week) Example: Count of Personal and Contact Information updates Count of Direct Deposit updates Count of Account Security updates Count of Connected Apps updates
Related to key result: Increased profile engagement/interactions
Thanks for your help with this!
cc/@Samara-Strauss @joanneesteban @jonwehausen @bsmartin-ep
☝️ I think we can close this issue once Matt's questions above are addressed.
@mshea0606 - Made the updates to the Profile 2.0 dashboard as requested. For the total interactions by page metric section, each one of the charts has the total interactions for each page. I can change that to only include updates for each page but wanted to get your input.
I know I said we could prob close this out, but I have another question — is it possible to set up a segment of people who have access to the Direct Deposit feature in the profile (so they can access the feature but don't necessarily have to have visited the page in a session)? We show this section conditionally based on whether you are eligible for direct deposit payments or not, so not everyone sees it.
Why I'm asking: I was pretty shocked to see that searches for "direct deposit" from the profile have actually not gone down since launch. In fact, they've gone up some weeks! This doesn't make any sense — direct deposit is clearly in the side navigation if you have access to the feature. There's no way people with access to the feature are missing it.
So, that led me to a new hypothesis: maybe the issue was not that we weren't clearly elevating direct deposit in the old profile, but that there were people who thought they should see this feature that were not seeing it. I'm trying to figure out a way to prove out this hypothesis. By looking at popular search terms, I can't tell whether the people initiating those searches can actually see direct deposit or not (I'm guessing no), but if I can determine that the reason people are searching is because they are not seeing that feature, then I've got something to move forward with. Let me know if you have any ideas on if we can set up a segment of some sort — whether it's people who have access to direct deposit (not sure that's possible) or something else. People who visit direct deposit actually may be an OK proxy.
And sorry, the above request is for a segment in GA — no need to update Domo now, though this might lead to a Domo update.
☝️ giving this a gentle nudge. Any thoughts?
Hi @Samara-Strauss - thanks for the nudge! Looking into this, I want to make sure I am understanding this correctly and have a quick question. Do all authenticated users have access to the Direct Deposit part of the profile? I assume this is true, but want to make sure.
Do all authenticated users have access to the Direct Deposit part of the profile?
No, they do not.
Let me know if you have questions about that.
So, basically I need a segment either of people who can see the direct deposit section, or if that's not possible, we can use people who update direct deposit and people who do not update direct deposit as proxies.
@Samara-Strauss - Here is a report showing "direct deposit" searches on profile
pages. There are two custom segements applied - 1 for authenticated users who updated direct deposit in the session and 1 for authenticated users who did not update direct deposit in the session.
Hey @bmcgrady-ep, am I missing something? I didn't see any segments in the report when I clicked that link:
@Samara-Strauss - I believe I need to share the segments with you:
Should I select the WBC view or All data view, or does it not matter?
Use the All Data view
I have a question: I checked out the results for the direct deposit segments mentioned above on searches made from /profile/personal-information (the first page of the profile) and was surprised to see that people who do update direct deposit and people who do not update direct deposit somehow don't make up 100% of searches from the profile. This is puzzling to me, because people either do or do not take this action from the profile; there's no middle ground. However, a third group — "all users" — rounds out the search counts (screenshots below) for searches made from /profile/personal-information.
I'm wondering if there is something wrong with the segments. For users who visit the profile, you either do or do not update direct deposit, and everyone should fit into one of those two categories. Why are we seeing "all users" round out the search count from the profile? What is that capturing that the direct deposit segments are not?
Let me know if you have questions. I can hop on a quick call if this is unclear.
@Samara-Strauss - I had the two segments set at session-level and have now switched them over to user-level segments. This more accurately segments the users into two distinct groups - 1 group who has updated their direct deposit and 1 group who has not updated their direct deposit. Here is the GA report with the two segments. Let me know if this helps!
Per discussion with @mshea0606, here are a few updates to the Profile 2.0 KPI dashboard to be made:
@bmcgrady-ep Would it be possible to also add "Total profile submissions" by week to help determine profile engagement?
@mshea0606 - Thanks for taking the time to meet with me. I added the Total Interactions on Profile
chart by week and the Total Profile Submissions
.
What is "Total profile submissions"? I have assumptions, but want to make sure I'm clear.
What is "Total profile submissions"? I have assumptions, but want to make sure I'm clear.
Total Profile Submissions are any profile saves. I can change the name if you and @mshea0606 have a different name you would like to use.
Thanks! That's what I assumed. No need for a name change.
@mshea0606 - I updated the KPI dashboard to show the number of ForeSee survey submissions by week. Let me know if there is anything else you need!
I'm going to close this. You all have been wonderful. We can open a new ticket if we have any additional requests.
Issue Description
Groom analytics for technical feasibility.
Tasks
Acceptance Criteria