search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
283 stars 204 forks source link

Analytics KPI dashboard request for Search & Discovery [Typeahead POC] #18785

Open denisecoveyduc opened 3 years ago

denisecoveyduc commented 3 years ago

Request a KPI Dashboard

What this form is for

Use this template to request a KPI dashboard on Domo for your product.

Instructions for the VFS team

Is this a new KPI dashboard request? Or is this an iteration? Please check the correct response.

VFS Information

KPIs

KPI Description Display Preference
Overall Volume Number of users for which type ahead feature flag was enabled (regardless of whether it was used or not) # of users broken out by month
Type Ahead Searches by Page Indicates from which page(s) type ahead searches have occurred/presented (regardless of whether they were chosen or not) Total # of searches by page by month
Type Ahead Usage A breakdown report of Type Ahead "Options" (suggestion 1 - 5) and "not set" (indicating a user ignored the type ahead options they were presented and proceeded to search organically (i.e. without selecting a suggestion). breakdown (% ignored + % engaged - indicated by suggestion #)
Top Type Ahead Options Selected An overall glimpse of the top keywords that were chosen from type ahead options list of keywords with corresponding # of times chosen by
View of experience without typeahead Comparing the two data points that highlight the amount to time it took to get from (search query) to (click on result) one with typeahead and one without. two line charts showing percentage of successful users over time (one with typeahead and one without typeahead)

Please leave the following blank

VSP Analytics & Insights Acceptance Criteria

  1. Groom Analytics

    • [x] Groom analytics for technical feasibility
    • [x] Document data sources for each metric
    • [x] Document unavailable vs. available metrics
    • [ ] If any, add implementation issue for additional necessary implementation

  2. Draft PST Dashboard

    • [ ] Draft dashboard on Domo
    • [ ] Structure data for ETL
    • [ ] Pipe available metrics to Domo
    • [ ] Add appropriate team members to dashboard

Please see checklist for full KPI dashboarding QA checklist

Definition of Done

jonwehausen commented 3 years ago

Hi @JasonTheBearded ,

Believe we should be able to deliver on each KPI Denise outlined above

KPI Description Display Preference Feasibility
Overall Volume Number of users for which type ahead feature flag was enabled (regardless of whether it was used or not) # of users broken out by month 🔶 GO
Type Ahead Searches by Page Indicates from which page(s) type ahead searches have occurred/presented (regardless of whether they were chosen or not) Total # of searches by page by month 🔶 GO
Type Ahead Usage A breakdown report of Type Ahead "Options" (suggestion 1 - 5) and "not set" (indicating a user ignored the type ahead options they were presented and proceeded to search organically (i.e. without selecting a suggestion). breakdown (% ignored + % engaged - indicated by suggestion #) 🔶 GO
Top Type Ahead Options Selected An overall glimpse of the top keywords that were chosen from type ahead options list of keywords with corresponding # of times chosen by 🔶 GO
View of experience without typeahead Comparing the two data points that highlight the amount to time it took to get from (search query) to (click on result) one with typeahead and one without. TBD 🔶 GO

Thought I would share with you a few starting queries I had from the VSP demo that start to answer some of the questions above:

joanneesteban commented 3 years ago

@denisecoveyduc - due to limited resources, we're looking to pull this into our next Sprint (2/4). Thank you for understanding~

denisecoveyduc commented 3 years ago

@joanneesteban Thanks that would be great! We have also defined one more that we discussed in a meeting with @jonwehausen. We would like to add it to this part of the dashboard. Let us know if you have any questions.

KPI Description Display Preference
Conversion Rate # of people who selected a typeahead suggestion and then selected one of the results from the results page line chart showing percentage of successful users over time
jonwehausen commented 3 years ago

Hi @denisecoveyduc ,

To follow-up from our conversation Wednesday 2/3, feel free to post your 1A KPIs for Type Ahead here and we will give them 1 more review to get started on the dashboard.

jonwehausen commented 3 years ago

cc: @JasonTheBearded for visibility

denisecoveyduc commented 3 years ago

@jonwehausen @JasonTheBearded This information is correct for the POC but I have narrowed down the metrics for #16778 which I am reviewing with the team later today for confirmation and then I will update that issue.

denisecoveyduc commented 3 years ago

@jonwehausen @JasonTheBearded Were you able to discuss this issue in your planning call this week? Any updates on expected delivery dates? Thanks!

jonwehausen commented 3 years ago

Hi @denisecoveyduc ,

Yes we were, this dashboard is in fact in in our Sprint 45.

cc: @joanneesteban @JasonTheBearded @mandy-uxd

denisecoveyduc commented 3 years ago

Excellent thanks @jonwehausen ! Newbie question but it sounds like your sprints don't align with ours since we are in sprint 40 ending on Feb 9. What is the end date of your sprint 45 and should we expect that the dashboard be ready for us on that date?

Also, here is the search.gov API endpoint URL. Let us know if you have any questions or need anything else.

Search.gov API endpoint URL

Information is under Getting Started on https://open.gsa.gov/api/searchgov-results/: The endpoint is https://api.gsa.gov/technology/searchgov/v2/results/i14y You must use https. You can find your access key on the API Access Key page of the Search.gov Admin Center.

cc: @JasonTheBearded

jonwehausen commented 3 years ago

Custom Metrics

@JasonTheBearded here is some brief documentation of the custom dimensions / metrics that were configured for type ahead

Custom Dimension / Metric Index Number Name Purpose
Custom Metric 39 Type Ahead Option Position Number to represent which type ahead option was selected from the list 1-5
Custom Metric 40 Search Results Position Number to represent what position in the search results was clicked on
Custom Metric 41 Search Results Total Pages Returned Number to represent how many total results were returned from search
Custom Metric 42 Search Results Total Pages Returned Number to represent how many search result pages were returned
Custom Metric 43 Search Results Click Page Number Number to represent what page the user was on when clicking on a search results
Custom Metric 44 Search Results Top Recommendations Increments by 1 if the search result clicked was a top recommendation
Custom Metric 45 Search Performed Redundant counter/incrementor for tracking search volume in addition to total events
Custom Dimension 114 Type Ahead Options Keyword Selected If the user clicked on a type ahead option, this will populate
Custom Dimension 115 Type Ahead Options List The full list comma-delimited of type ahead options at time of search
Custom Dimension 116 Phased Out Product Can look for type ahead here or at the event level
Custom Dimension 117 Type Ahead Option Chosen Position Redundant collection of type ahead chosen position
Custom Dimension 120 Search results type Determines the type of search result that was clicked on, everything from type ahead should be "title"
JasonTheBearded commented 3 years ago

@denisecoveyduc For the KPI 'Type Ahead Searches by Page' there are quite a few pages from which searches are occurring. Would it be helpful to either display the top pages or group the pages to reduce the number of results?

denisecoveyduc commented 3 years ago

@JasonTheBearded Without seeing an example I am not sure how we would group them so how about we start with the top pages since those are what we will really want to know anyways.

jonwehausen commented 3 years ago

QA of Dashboard

Items for further examination

Validated

✔️ Type Ahead Usage by Options ✔️ Type Ahead Options Selected ✔️ Timing between search + search results click

Questions 💭

image

JasonTheBearded commented 3 years ago

@jonwehausen after reviewing denise's comment on my question in slack, I believe you are correct on adding a condition to compare type ahead conversions against those who had type ahead options. I'll be making that change.

denisecoveyduc commented 3 years ago

@jonwehausen @JasonTheBearded I do love the "beast mode" terminology. 🤪 Also love the progress, thanks! You are correct we were thinking conversion rate on this issue is only for the users where a typeahead option was chosen.

jonwehausen commented 3 years ago

Thanks for your confirmation here @denisecoveyduc !

joanneesteban commented 3 years ago

@denisecoveyduc we're wrapping on a couple of things and should be able to get this to you this week. Can you let us know when you might be available this week or early next week so that we can walk you through the dashboard?

@jonwehausen in @JasonTheBearded 's absence, would you be able to add:

@mandy-uxd if we can, let's look into structuring this around the framework so we don't have to tweak later.

jonwehausen commented 3 years ago

Google Analytics report links have been added in the bottom appendix section where applicable + possible.

denisecoveyduc commented 3 years ago

Hi @joanneesteban We'd love to touch base this week if at all possible. Let me throw out a few times, see if any of them stick. Thanks!

1-Wed Feb 17: 4 - 4:30pm EST 2-Thursday Feb 18th: 12-12:3pm EST 3-Thursday Feb 18th: 2-2:30pm EST 4-Friday Feb 18th 12-12:30pm EST

denisecoveyduc commented 3 years ago

February 19 - 12-12:30pm EST invites sent to walk through dashboard

joanneesteban commented 3 years ago

For the VSP team: notes to transition from Jon back to Jason

JasonTheBearded commented 3 years ago

@mandy-uxd Type Ahead dashboard data changes complete, ready for design review. Thanks!

denisecoveyduc commented 3 years ago

Thanks to @JasonTheBearded and @jonwehausen for a final touchpoint. As discussed we identified a few final revisions for the dashboard which I wanted to document here. I will establish a new request for enhancement to add bounce and refinement rate comparisons. Let us know if there are any questions or concerns. Thank you!

Actions:

FYSA: @joanneesteban @johnhashva

JasonTheBearded commented 3 years ago

@denisecoveyduc I've made the requisite changes to the dash so feel free to give it a look. I renamed the "Users with TypeAhead enabled and selected a typeahead option" chart to "Type Ahead Users and their engagement with Type Ahead".

Regarding that chart I have a question. After adding the second bar for users who did not select an option, I'm thinking it's possible that a Type Ahead user could conduct a search and use one of the options, then come back and perform another search and not use an option. In this scenario that user would get counted twice. So instead of # of users would it be better to display # of searches?

denisecoveyduc commented 3 years ago

@JasonTheBearded That is an excellent point, and I think you are right! Thanks for catching that! Let's go ahead and change it to the # of searches as you suggest - that is most meaningful to our decision making.

Love the addition of the total searches by page that is nice to see! Just to confirm my understanding about the totals, does this screenshot and my orange comments depict it correctly? I assume that once we are at 100% those numbers will be the same.

Screen Shot 2021-03-03 at 10 25 51 AM
JasonTheBearded commented 3 years ago

@denisecoveyduc Thanks for your patience! I made the change to the Typeahead engagement to reflect searches instead of users. When I was looking at those numbers I noticed that we weren't capturing instances where a user has typeahead enabled, performs a search, but does not receive any typeahead options in the graph. I have added this group of searches to the graph which should portray a more accurate picture of typeahead searches.

To answer your question about the screenshots. The first screenshot for "total # of searches conducted when typeahead is enabled" is correct, that number will be more accurate now that we've added the third group. For the second screenshot, that number is the total # of searches conducted by users who have typeahead enabled, regardless of engagement. It does not include users who do not have typeahead. This number will be lower than the above graph because it is only including the top pages rather than all.

Let me know if you have any additional questions. Thanks!

denisecoveyduc commented 3 years ago

Thanks @JasonTheBearded I will review the recent updates with the team today and let you know if there is any feedback.

Also FYI, I did create the enhancement ticket we discussed to add bounce and refinement rates GH #20838 We know you have a lot on your plate, let us know when you think you might be able to fit this in. Thanks again!

cc: @joanneesteban

joanneesteban commented 3 years ago

Moving to blocked on the VSP project board because of the lack of data.

JasonTheBearded commented 3 years ago

Hey @denisecoveyduc! I wanted to let you know that we've pushed the date forward for our Type Ahead datasets to 3/23/2021 so the dashboard will no longer have any historic data, except for our monthly and daily user counts. We left those in because the type ahead enablement is separate from the type ahead suggestion issue, but if you find it confusing to see those numbers and the small search data returned, we can certainly push the date forward on those cards as well.

Two additional items came up that we were hoping you could help provide answers on. 1) Do you know when the additional suggestions would be making their way into the system? Having an idea on that will help us keep an eye on the feed and ensure the cards get updated accordingly.

2) With our search data layer now scaling to other products, we wanted to confirm on charts where we're comparing type ahead enabled vs disabled: should our "type ahead disabled" searches compare against all other va.gov search products (that have type ahead disabled) or should searches be considered only from products that have type ahead enabled?

Thanks and I appreciate your patience on this! cc: @jonwehausen

denisecoveyduc commented 3 years ago

Hi @JasonTheBearded. It appears that I missed this comment (apologies). We have been busy creating a list of suggestions to hardcode onto the database and are now ready to implement typeahead once again. We plan to run a second POC starting on 4/12(Plans are outlined on #21316)

Would you be able to push all of the date forward to April 1st, 2021? I think that the monthly and daily user count data should also be removed and to eliminate the data that shows up in March it would be great to just bump everything to April.

To answer your other question the only reason we have the "typeahead disabled" category was because for our POC we are only opening it up to a small subset of our user base (POC 1 was 5%; POC 2 will be 35%) As a result not all users had typeahead suggestions presented to them. Hopefully that helps.

joanneesteban commented 3 years ago

Thanks for the response and questions, Denise! The Platform Analytics team is pausing on doing custom Domo and GA reporting and dashboarding until June, so any new or open tickets will remain in the backlog and we will reach out with an estimated timeline then. Please see this announcement for more details.

Additional analytics self-service documentation can be found in the: Analytics Playbook KPI Framework Google Analytics folder Google Analytics Cheat Sheet Google Analytics FAQ Google’s Google Analytics training

denisecoveyduc commented 3 years ago

Hi @joanneesteban Yes I was aware of the pause and really don't need anything except for the data change if that is at all possible. The dashboard is set up and ready to go, we just wanted to remove the data from before April 1st as we are getting ready to turn the feature on officially on Monday. IF it is possible to switch the date to April 1st that would be great. If not we will deal with what we have today. Good luck on the infrastructure project - hope things are going smoothly!