search

department-of-veterans-affairs / va-mobile-app

"If VA were a company, it would have a flagship mobile app."
https://department-of-veterans-affairs.github.io/va-mobile-app/
17 stars 2 forks source link

BUG - sev-2 - Android - Seems like Android devices get RC flag updates less than iOS devices do #5271

Closed TKDickson closed 1 year ago

TKDickson commented 1 year ago

What happened?

Generally, our Android/iOS split is ~35% Android, and ~65% iOS. And you can see this across feature usage, for instance viewing appointment details, which is 30% Android traffic: image.png

But for features that are still tied to feature flags, Android stats lag significantly (closer to ~10-15%/~85-90% split for Android and iOS). Here's encouraged update being shown to users, which has been out for a while and just 13% of all traffic is Android: image.png

Here's Rx refill success - Android is 8% of all traffic: image.png

I'm not sure what's going on here, but it's worth looking into to make sure we're not inadvertently excluding Android users from new features for longer with how RC/feature flags work.

The lower stats for Android across the board (in events) are also echoed by the lower % of successful Android fetches, where firebase defines fetch as "a successful request from your app to the Remote Config server". Hovering at 10% Android for both of the above features: image.png

Specs:

Steps to Reproduce

To get to these various dashboards: Go to firebase

Fetch rate/RC stuff is in "Remote Config" (lefthand sidebar, in "Project Shortcuts" section). It's automatically separated by iOS vs Android, you just need to tap the row to expand the RC you're interested in.

All of the other graphs shown are events-based, so go to "Events" (from Project Shortcuts section of left sidebar), then filter using the magnifying glass to just the event you're interested in, then tap on the row name to open an event-specific graph like the ones I screenshotted above. To show iOS vs Android, you'll need to tap the "Add Comparison" button (by the blue "all users" label tag near the top of the screen), then use the right sidebar that pops open to complete the workflow - select "platform" for dimension, then either iOS or Android for dimension value, then tap apply. Then do that again, but pick the other one (iOS if you already picked Android). image.png

Desired behavior

Acceptance Criteria

Bug Severity - BE SURE TO ADD THE SEVERITY LABEL

See [Bug Tracking](https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/va-mobile-app/testing/VA%20Mobile%20App%20Test%20Plan.md#issue-severity) for details on severity levels

Linked to Story

Screen shot(s) and additional information

Ticket Checklist

alexandec commented 1 year ago

@TKDickson This is interesting, looking at a longer time period, I can see that Android's percentage was lagging, but over time it has increased:

vama-eu-shown.png

I see the same when checking the Remote Config stats:

remote-config.png

The slow uptake for Android is probably not caused by devices caching Remote Config flags, because we have set our Remote Config client to use a fetch timeout of 10 seconds and a cache time of 30 minutes. Those settings are the same on both platforms.

My best guess is that Android devices aren't automatically updating apps as frequently as iPhones are. When an app update does happen, the device should kill the app if it's already running, then install the new version. When the user runs the updated app, the app should immediately fetch the latest remote config values at that point.

I did some research and couldn't determine anything further about why Android might apply automatic app updates more slowly than iOS. I wonder if the App Store and the Play Store could provide us with data showing how many users have each particular app version installed, or available to be installed, over time. That could help determine whether app updates are at the root of the problem. Binny mentioned we've seen updates being slower to roll out on Android compared to iOS.

This concludes my quick investigation. I'll increase the estimate on the ticket in case we want to do more in the future.

alexandec commented 1 year ago

@bischoffa I did a quick investigation on this bug, wrote up my findings, updated the estimate, and moved it to the backlog

Sparowhawk commented 1 year ago

@bischoffa I don't have access to firebase or the individual stores to check version deployment and overall app update rate that may be contributing to this issue. However I did find a few things when looking at Android specific things.

  1. Android users are more likely to turn off automatic updates(might be contributing)
  2. Users may not be on wifi for their updates(default is to wait for wifi to start updating apps and then they may be in a queue for ours)
  3. Google play store is slower to deploy on average and large changes in What's New content can apparently delay the release to users and so can changes to permissions(don't think we had any of those lately)
  4. Unfortunately there is no way to determine what the users settings are and we currently do not have a way to determine if the user is on wifi or not to help determine this. If I had access to the stores and firebase I might be able to figure out more. @timwright12 maybe you can assist with giving me access?
bischoffa commented 1 year ago

@kellylein fyi as this is a bug with blue team 

@Sparowhawk @alexandec thanks for the details and your write ups. 

I pulled this out of the sprint as we do not have capacity to add more points to FE. Plus seems like additional conversations / discussions needed based on the findings.

bischoffa commented 1 year ago

@kellylein Binny reched out to me about this ticket, I asked him to touchbase with you on it.

dumathane commented 1 year ago

Closed per @kellylein and @timwright12 .