[Eperiment] No "messaging.shown" telemetry event is sent after the "New User Survey" is triggered on iOS devices

[data-sync-user]

[Affected versions]:

• Firefox Release 126.0 (41006).
• Firefox Release 125.0 (40715).

[Affected Platforms]:

• iOS 16.2 - iPhone 12 Pro.
• iOS 15.8 - iPad Air 2.
• iOS 17.4 - iPhone 15 Pro (simulated device)

[Prerequisites]:

• The user is enrolled in the “New User Survey iOS – April 2024 (125)” experiment if Firefox Release > 125.0 is used; or in the “New User Survey iOS - April 2024 (122-124)” experiment if Firefox Release 122 - 124 is used instead.

[Steps to reproduce]:

1. Open the Firefox app.
2. Force quit the Firefox app and open Safari.
3. Navigate to "firefox://glean?logPings=true&sendPing=events&debugViewTag= and tap the "Open" button.
4. Navigate to https://debug-ping-preview.firebaseapp.com/pings/ on any device and verify the registered events.

[Expected result]:

• The New User Survey is triggered and a "+messaging.shown+" telemetry event is displayed on the "Glean Debug Ping Viewer" page.

[Actual result]:

• The New User Survey is triggered, however, there is no "+messaging.shown+" telemetry event displayed on the "Glean Debug Ping Viewer" page.

[Notes]:

• This issue is not reproducible on the Android version of the experiment.

┆Issue is synchronized with this Jira Bug

[data-sync-user]

➤ Jon Stermac commented:

Marius Coman can you confirm that this issue is present in both experiments, the >125 and the the 122-124 one?

[data-sync-user]

➤ Marius Coman commented:

Hi, Jon Stermac! I can confirm that this issue is reproducible on Firefox Release 122.2 (38550) and Firefox Release 124.3 (40336) builds installed from TestFlight while enrolled in the New User Survey iOS - April 2024 (122-124) ( https://experimenter.services.mozilla.com/nimbus/new-user-survey-ios-april-2024/summary ) experiment. The “message_shown” telemetry event is not sent, however, for both experiments the “clicked” and “dismissed” survey events are successfully displayed on the “Glean Debug Ping Viewer” page.

[data-sync-user]

➤ Jon Stermac commented:

Thank you Marius Coman , so it sounds like some of the telemetry events are being sent but not the main “message_shown” one.

We’re trying to isolate if this is an issue with some changes that were deployed for the experimentation platform that affect Firefox Release 125. Are you able to run a test (if you haven’t already) to see if the same issue is present when using Firefox Release 125 with the New User Survey iOS - April 2024 (125) ( https://experimenter.services.mozilla.com/nimbus/new-user-survey-ios-april-2024-125/summary ) experiment?

[data-sync-user]

➤ Marius Coman commented:

Hi, Jon Stermac! Yes, the issue is reproducible on both the "New User Survey iOS - April 2024 (122-124) ( https://experimenter.services.mozilla.com/nimbus/new-user-survey-ios-april-2024/summary )" and "New User Survey iOS - April 2024 (125 ( https://experimenter.services.mozilla.com/nimbus/new-user-survey-ios-april-2024-125/summary )" experiments with the Firefox Release 126.0 (41006), Firefox Release 125.0 (40715), Firefox Release 124.3 (40336), and Firefox Release 122.2 (38550) builds installed from TestFlight. We have successfully managed to generate and track a "messaging.shown" telemetry event coming from the "default-browser" message which is on by default in the browser.

[data-sync-user]

➤ Jon Stermac commented:

Thank you for confirming!

[data-sync-user]

➤ Jon Stermac commented:

Marius Coman can you confirm that you are not seeing the “messaging.shown” telemetry for these survey experiments? Cyndi Chin mentioned that iOS uses “messaging.shown” instead of “message_shown” on Android. Thanks!

[data-sync-user]

➤ Marius Coman commented:

Jon Stermac thanks for investigating this behavior! Yes, I can confirm that I'm not seeing the “messaging.shown” telemetry event from the "survey" message surface.

[data-sync-user]

➤ Jon Stermac commented:

Thank you for confirming.

Tagging Norberto Andres Furlan as this is considered a blocker for this experiment to go live.

[data-sync-user]

➤ Roux Buciu commented:

Looking at this, I can verify that we’re sending a telemetry event correctly.

The parent trigger is with message-impression when the message is displayed (and the code path gets executed)

!Screenshot 2024-04-26 at 10.52.33 AM.png|width=756,height=226,alt="Screenshot 2024-04-26 at 10.52.33 AM.png"!

This sends a Glean telemetry event under messaging . The event is shown (as per our metrics structure)

!Screenshot 2024-04-26 at 10.54.16 AM.png|width=637,height=363,alt="Screenshot 2024-04-26 at 10.54.16 AM.png"!

This can all be seen in the glean dictionary entry ( https://dictionary.telemetry.mozilla.org/apps/firefox_ios/metrics/messaging_shown ). I’ve verified that this gets to the Glean pipeline. If Glean is not sending it, that’s a whole other problem. But, to summarize, message_shown is not expected because we don’t have anything that sends that telemetry. Rather, looks like messaging.shown is what’s expected.

[data-sync-user]

➤ Marius Coman commented:

Hi Roux Buciu, thanks for investigating this behavior, and sorry for the confusion! Initially I was under the impression that we have the same naming convention as on Android, that’s why I’ve written the issue using the “message_shown” impression. However, we can confirm that we are not seeing the “messaging.shown” telemetry event on the “Glean Debug Ping Viewer” page either.

I will update the bug to contain the correct event name.

In order to further investigate this behavior we created a +Stage recipe+ ( https://stage.experimenter.nonprod.dataops.mozgcp.net/nimbus/mcoman-viewpoint-1-week-expansion-test-impression-testing/summary ) containing a survey message that was part of the “Viewpoint 1 Week Expansion Test” experiment which targeted Firefox 112. We can confirm that using the Treatment A branch of the “Viewpoint 1 Week Expansion Test” experiment on Firefox 114.4 we have managed to trigger a “+messaging.shown+ ( https://debug-ping-preview.firebaseapp.com/pings/cmuresan-29/4eaf19b4-e997-4945-ae3a-e4dc94c4047e#L62-L70 )” telemetry event using the same steps as for the “New User Survey iOS - April 2024” experiment.

Do you see any value in searching for differences between these two builds that could lead to this behavior? Also, do you know who we could ping from the Glean team to give us their opinion regarding this?

[data-sync-user]

➤ Marius Coman commented:

Roux Buciu we found out that the messaging events are sent and logged on Looker, but for some reason, we are not able to tag those events using Glean. Here is the Looker query: https://mozilla.cloud.looker.com/explore/firefox_ios/event_counts?qid=gjaZHTkefntM17Qlj5NW5F&toggle=fil,vis ( https://mozilla.cloud.looker.com/explore/firefox_ios/event_counts?qid=gjaZHTkefntM17Qlj5NW5F&toggle=fil,vis ).

[data-sync-user]

➤ Marius Coman commented:

We have retested this issue and we can confirm that the telemetry pings are successfully sent to the server. 

It seems that we hit a Glean limitation during testing and the telemetry events are not tagged if the browser is sent to the background right after the survey was displayed and no other action was taken. However, if any survey button is tapped right after the survey is triggered, the “messaging.shown” telemetry event will be successfully tagged and sent to the “Glean Debug Ping Viewer” page.

Considering the above I will close this issue.