Analytics: compare findability data for BTSSS & supply tools between mobile and desktop layouts

[wesrowe]

Description

User story

As a Cartographers team member, I want to compare the usage of supply reordering and BTSSS links between mobile and desktop devices, so that I can see if the mobile layout/length of scroll of the MHV landing page might be impeding discovery and use of these tools.

Notes

• Assumptions
  • Datadog implementation (in our codebase) isn't necessary, as DD already has facets for device

Possible tasks:

Acceptance criteria

• Segmented clicks on BTSSS and supply-reorder links by layout (desktop vs. mobile) so that we can see if either layout is significantly more likely to get a click on each of these links – data broken out per-link and aggregate
• Sought device segmentation of the apps themselves (probably in GA4), to see if there is a device disparity at the links' destinations
• Share findings with Patrick
• Ticketed next steps
  • if no difference, maybe nothing?
  • possibly set up DD so we can segment just users who are eligible to reorder supplies online
  • run an experiment if appropriate

[LynneJohnson224]

@jonathanjnelson can you please add a comment as to why this ticket is blocked? thanks!

[fmccaf1]

I think I got to the bottom of clicks for BTSSS. Here's the data I pulled using these reports in GA4 for June 1-30, 2024:

Percent desktop views of my-health and my-health/: 62.7% Percent mobile views of my-health and my-health/: 36.6% this seems right, around 30% mobile users

Percent desktop clicks on BTSSS: 69.5% Percent mobile clicks on BTSSS: 28.4%

So assuming this data is right (which may be the wrong assumption), it does look like there are more clicks coming from desktop users. Is this because of the long scroll on mobile? Or something else?

@katbrinkley @jonathanjnelson curious if you have thoughts ^

Edit: I just did a calculation and this difference is small enough that it is not statistically significant even at a 90% confidence interval. Let's keep watching these numbers.

[fmccaf1]

Learning GA4 on the fly here but wondering if it's enough to look at event count / user segment to give us an idea as to whether the event is happening at the same rate on desktop as it is on mobile. If yes, it seems that medical supplies links are also being found at pretty similar rates on desktop and mobile:

[fmccaf1]

@batemapf if you're curious, see above. It is looking so far from analytics that mobile users are not having too much trouble finding the links at the bottom of the page.

[wesrowe]

@fmccaf1, I'm interested in the statistical method you applied. >8% difference for a month of data seems pretty likely to be real – but of course statistics were invented to overcome lousy intuition. (Whether the difference matters is another story.)

[fmccaf1]

@wesrowe I just found and used one of the many statistical significance calculators online. I think I used this one (qualtrics).

[katbrinkley]

wow, nice Florence! I feel like this is the segmenting we wanted in the ticket that I was going to try in datadog. do we still think we should track BTSSS in its own widget, or does this satisfy what we needed?

[wesrowe]

@fmccaf1, thanks for sharing the link to the calculator. I followed your trail to see if I could reproduce your result. Since it's a conversion I needed something like total page visitors, which I didn't see in your comment above so I could have diverged there. (See below.)

I grabbed data for the past 90 days. The inputs are calculated in this GA4 exploration, see the BTSSS tab [note you can see the time period etc in the screenshot]:

User stats for same time period (first tab in same GA4 exploration):

I'm not sure whether to use Total Users, Active Users, or Views for the conversion calc. I went with active users (which is basically the same as total users).

I got the "not significant" result from the Qualtrics calculator down to 90% confidence, just like Florence:

However, the calculator at Survey Monkey says it's significant based on the inputs from GA4 above, up to 99%(!):

Why the discrepancy between two calculators? I prefer the SM one because it agrees with my intuition. ROTFL

I don't know this stuff beyond a high school level. But I noticed that the conversion description on the SM tool talks specifically about the ratio of the conversion rates being significant – i.e., mobile users are 33.3% less likely to "convert" (click the BTSSS link). I find this at least provides a little insight into the "thinking" of the calculator.

@rbsingleton and @katbrinkley, anyone have deeper knowledge of how statistics apply here? The goal, in case it's not obvious, is to see if mobile users are significantly less likely than desktop users to click the BTSSS link. Florence was the one who thought of applying actual statistics.

[fmccaf1]

I'm still working on this @wesrowe, but one thing of note: it looks like the dates you entered are not the same as those I entered for this data, so you likely have a lot more data points (which is often what is missing when statistical significance is not met).

Should we be excluding the time period to only June 17 and later?

Another thing I'm noticing, the second calculator you used provides the option for a two-sided hypothesis vs. a one-sided and looks like two-sided is chosen. This is confusing and strange, because a two-sided hypothesis is more conservative actually, so statistical significance is harder to reach with a two-sided hypothesis (you'd need more users total or a more clearly split favorite. On top of that, the confidence level is 99%! So again, making it even harder to prove significance because now we're saying we're 99% the results are significant as opposed to what I calculated, which would make us 90% sure.

I'm going to do some work into looking into two-sided hypothesis and if we need to be using that calculation instead.

[fmccaf1]

To increase statistical significance with high confidence levels, I used more data (further into the past): Apr 17 - Jul 29, 2024. I pulled data from two GA4 reports: MHV home views and Link clicks.

I added all data that I pulled to this excel spreadsheet in our Sharepoint folder.

In summary, I found statistical significance at 99% confidence that desktop users are more likely to click the following links: BTSSS, pay copay, order CPAP supplies, order hearing aid batteries.

One concern I have about this data: I don't know why in the MHV home views report, there are no desktop sessions recorded for /my-health. See screenshot:

[wesrowe]

@fmccaf1, I think I figured out why you have zeroes. The first zero at the left is saying, among pages located at /my-health/, there were zero users viewing /my-health. (I.e., with and without the trailing slash.)

However, the possibility you raise of a mobile disparity for the no-trailing-slash version of the URL turns out to be very interesting anyway! I found a clear skew in /my-health toward mobile devices – i.e., the trailing-slash version of the url is disproportionately high on desktop, whereas on mobile the version without trailing slash is almost at parity. Fascinating! I'll post to the Analytics team in slack.

[wesrowe]

Update: asked Jamie in slack, she doubted my device hypothesis. I.e., "devices/browsers don't change anything."