search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
283 stars 204 forks source link

Examine and Summarize Typeahead POC data #19693

Closed denisecoveyduc closed 3 years ago

denisecoveyduc commented 3 years ago

Description

Typeahead POC launched on December 23, 2020 - January 31, 2021. The analytics team is working on establishing a dashboard to showcase typeahead data (GH#18785) and is expected to be completed in their sprint # 45 which ends on February 16th. Once the dashboards has been established a final review of the data needs to be conducted.

Background/Context/Resources

Tasks

Definition of Done

denisecoveyduc commented 3 years ago

Due to refinements needed on the Typeahead dashboard we need to bump this issue to the next sprint.

denisecoveyduc commented 3 years ago

POC (December 23- January 31) Conclusion Summary

POC Summary The initial POC for typeahead was expected to run for 4-6 weeks with a user base of 5% of all va.gov users after which time the plan was to assess the data collected and determine how effective the typeahead feature was at improving the experience for veterans. A variety of factors (explained in more detail below) have blocked our ability to interpret the data from the POC and as a result we have decided to declare this POC inconclusive and reset the dates for a second POC which we believe will be more representative of the actual typeahead feature and its implications.

Search.gov Suggestions Database Purge Search.gov resolved a bug fix which was preventing suggestions older than 30 days from being purged from the database. This bug as well as the purge have distorted the suggestions database to the point that conclusions can not be drawn because of a lack of high quality/ targeted suggestions that improve the users ability to find relevant results.

Analytics Implementation Various analytics discoveries and changes due to implementation changes or accessibility fixes created an environment where the data has been distorted and not easily interpreted or used for definitive conclusions.

Conclusions Engagement: We did see a large percentage of user engagement (approximately 42.8%) with the feature showing appetite for the experience, despite suggestion relevance being a negative factor. Month Total # of users who had typeahead available to use Used Typeahead Did not use typeahead
December 2020 2524 1046 1478
January 2021 13081 5638 7443
TOTALS 15605 6684 8921

[CAVEAT: This data is what was presented on our DOMO dashboard; however given the analytics implementation discussed above we are making some assumptions knowing that this data may be inaccurate]

Importance of Quality Suggestions: We discovered that the quality of a typeahead suggestions played an important factor in the users ability to find relevant results. Our hypothesis is that suggestions that added irrelevant additional phrases and words lowered the likelihood of a user getting a relevant result as a top result and lowered the likelihood of the user getting a best bet. This contributed to a lower-than-average click-through rate and we believe that incorporating suggestions relevant to the VA will improve the users ability to find relevant content.

Sample Size: We believe that a 5% sampling of users is too small of a sample pool to get accurate user interaction results. On average only 10% of visitors use search, and we believe that releasing to 25% of users will give us more consistent usage data.

cc: @johnhashva, @mgleo, @srsuddath