search

mozilla-mobile / mozilla-vpn-client

A fast, secure and easy to use VPN. Built by the makers of Firefox.
https://vpn.mozilla.org
Other
453 stars 108 forks source link

Some Subscription related metrics were not implemented #8975

Open data-sync-user opened 8 months ago

data-sync-user commented 8 months ago

As reported by [~accountid:60ec9eaaf314ff006a44bb42] on Slack, these are the missing metrics:

Looks like oversight.

Miro board:https://miro.com/app/board/uXjVM0BZcnc=/

Update 5/3/2024: [~accountid:60c36c45dae567006809d3e2] [~accountid:634d8e68fe5ff375235781b3] and I discussed and agreed on the following:

┆Issue is synchronized with this Jira Bug ┆Reporter: Beatriz Rizental Machado

data-sync-user commented 7 months ago

➤ Beatriz Rizental Machado commented:

re: interaction.manage_subscription_selected This one is in there. I can even see some counts in Looker: https://mozilla.cloud.looker.com/explore/mozilla_vpn/event_counts?qid=vzNQGs1XvAEI1J2oS55StG&toggle=fil,vis ( https://mozilla.cloud.looker.com/explore/mozilla_vpn/event_counts?qid=vzNQGs1XvAEI1J2oS55StG&toggle=fil,vis ) – very low numbers though.

data-sync-user commented 7 months ago

➤ Beatriz Rizental Machado commented:

re: The subscription outcomes. As I add these metrics I notice there are some missing outcomes in there, here they are:

subscription_failed: type: event lifetime: ping send_in_pings:

data-sync-user commented 7 months ago

➤ Beatriz Rizental Machado commented:

re: The confirming_subscription one. That screen doen’t seem to exist in the app anymore?

data-sync-user commented 7 months ago

➤ Santiago Andrigo commented:

Beatriz Rizental Machado

data-sync-user commented 7 months ago

➤ Beatriz Rizental Machado commented:

{quote}I don’t see any confirming_subscription on neither the old telemetry miro ( https://miro.com/app/board/uXjVMVLwsKo=/ ) nor the new one ( https://miro.com/app/board/uXjVM0BZcnc=/ ). I also don’t see any events on Looker ( https://mozilla.cloud.looker.com/explore/mozilla_vpn/event_counts?qid=yShncLFtfXiMIaoxbltski&toggle=fil,vis ) in the last 180 days, so I’m guessing that whatever screen this corresponded to, it’s no longer there.{quote}

On the new one there is such a screen. See the mobile flow.

{quote}subscription_not_validated: what does it mean? can we have a more human readable / intuitive error name?{quote}

This happens when the subscription attempt fails, not sure what makes this different from the subscrition_failed event but there s a distinction in the code.

data-sync-user commented 7 months ago

➤ Santiago Andrigo commented:

Ah sorry, I was looking for the outcome events, but this was just a regular one. Gotcha. Are you saying that that screen (“You are all set”) no longer exists?

When, in the Miro flow, would subscription_failed go in? Want to make sure we are tracking there as it becomes the easiest way for somebody not looking at the code to know what’s available for telemetry.

data-sync-user commented 4 months ago

➤ Valentina Virlics commented:

Currently (latest 2.23), if we use the Telemetry debugging (Dev tools) option to test this, every time we sign out, the tag is removed.

If we add the tag back, and click the “Cancel” button while in the “waiting to log in on the web“ client state, the tag is removed as well. (see ss below)

!image-20240515-121509.png|width=1026,height=785,alt="image-20240515-121509.png"!

We are not sure how we could test this? Are we missing something in the setup process? Thank you!

Santiago Andrigo Beatriz Rizental Machado

data-sync-user commented 4 months ago

➤ Beatriz Rizental Machado commented:

Hm, I wonder why the tag is removed on logout. I will check it out. Can you test it by just logging the pings instead of sending them to the debugviewer for the time being?

data-sync-user commented 4 months ago

➤ Beatriz Rizental Machado commented:

Another workround would be to set GLEAN_DEBUG_VIEW_TAG="a-tag" in the environment the application is running, that should work. Let me take a look first though.

data-sync-user commented 4 months ago

➤ Valentina Virlics commented:

Beatriz Rizental Machado I can try, but how can I log the pings? 😳 And how to set the GLEAN_DEBUG_VIEW_TAG="a-tag" in the environment the application is running? Thank you!

data-sync-user commented 4 months ago

➤ Valentina Virlics commented:

I managed to have a quick call with Bea in order to understand how to set a tag via Terminal. Thank you, Bea!!:smiling_face_with_3_hearts: We created the tag with success!

After making the following steps:

The tag/pings are visible in Glean, but no events like outcome.subscription_started/outcome.subscription_completed or outcome.subscription_failed (with specific reasons) are displayed.

!Screenshot 2024-05-17 095417.png|width=802,height=234,alt="Screenshot 2024-05-17 095417.png"!

Thinking into this, are we sure that - while on desktop the In App Auth is moved outside the app - should we still see those subscription events?

Beatriz Rizental Machado Santiago Andrigo

data-sync-user commented 4 months ago

➤ Beatriz Rizental Machado commented:

Yes, we should still see the events even with browser subscription. There must be some implementation or testing bug 🤔

data-sync-user commented 4 months ago

➤ Beatriz Rizental Machado commented:

Huh, ok. I tried it out and I also don’t see the outcome events. We might need to put this back in progress. I wonder why the tests pass 🤔

data-sync-user commented 4 months ago

➤ Beatriz Rizental Machado commented:

Ok, the events are only recorded when we get to the “Subscription Needed” view. If we never get there there is no way for the app to know if the user is just logging in or subscribing. Therefore, you are correct Valentina Virlics , with authentication moved ton in browser I think don’t have reliable way to record these events. I have only briefly looked at this, maybe Lesley Norton can take a look later on as she knows more about in browser auth.

data-sync-user commented 4 months ago

➤ Valentina Virlics commented:

Lesley Norton We should either move this back to Backlog, if we want to fix this in the future, or close it as Cancelled, as currently (with web login) those events are not visible.