defagos commented 3 years ago

The comScore SDK emits a start event when initialized. This event must only be related to the user proactively starting an application, not if the application is started silently in the background (by a push notification, visible or silent, most notably).

Our SDK does not make a difference between these startup scenarios. Our documentation mentions the SDK should be started in the application:didFinishLaunchingWithOptions: method, but we don't check whether this method is called as a consequence of the user launching the app or without any interaction.

This is blocker for Mediapulse, since it creates false signals which are not acceptable.

How to reproduce the issue

First we need to reproduce a ghost start. My understanding is that it should be possible as follows:

Monitor networking with Charles on a real device.
Kill the app.
Send a notification to the app.
Check if a start event is seen in Charles.

If a start event is seen we have a reproducible test scenario.

App lifecycle

We must verify the app lifecycle, in particular when the application:didFinishLaunchingWithOptions: can be called non-interactively.

Fix

If this is related to the application lifecycle, we must defer SDK startup in non-interactive cases, which probably means:

When the start method is called while the app is not in foreground (check the UIApplication state), then only save the startup configuration parameters but do not initialize the SDK.
When the app moves to foreground use the saved startup configuration to start the SDK.

defagos commented 3 years ago

It is likely that ghost starts can be seen if SRG Analytics is used in widgets, which is something we don't support officially yet. Either:

This can be fixed in a similar way.
This requires more work (and probably a different enhancement issue).
This requires better API (e.g. unavailability attributes for extensions) and documentation (e.g. SRG Analytics not supported in extensions yet).

defagos commented 3 years ago

I applied the procedure outline above (kill the app and trigger a notification while checking network traffic). I cannot see any start even as a result of the push notification being received, not even after waiting for quite some time.

Also I could not associate analytics with the SRF Meteo widget.

Back to square 1 😢

defagos commented 3 years ago

I just got the issue with Play RSI beta, which was not running on my device (not even in my empty task switcher). When the notification appeared on screen I immediately saw the ghost start event:

:method: GET
:scheme: https
:path: /p2?c1=19&c2=6036016&ns_ap_an=Play%20RSI&ns_ap_pn=ios&ns_ap_pv=14.5.1&c12=7CC327C611839B7C7B87A938161ED874-cs62&name=background&ns_ap_ec=1&ns_ap_ev=start&ns_ap_device=iPhone11%2C2&ns_ap_id=1621246495695&ns_ap_csf=1&ns_ap_bi=ch.rsi.rsiplayer.beta&ns_ap_pfm=ios&ns_ap_pfv=14.5.1&ns_ap_ver=3.1.9-beta&ns_ap_sv=6.7.1.210208&ns_ap_bv=2.6.1.210205&ns_ap_smv=6.5&ns_type=view&ns_ap_gs=1505459285199&ns_ts=1621246490671&ns_ap_cfg=110000100-001-3C-7D0-64-A-1F-1E-BB8-BB8&ns_ap_env=0-1-0-0-2-0&ns_ap_ut=60000&ns_ap_ar=arm64&ns_ap_cs=354&ns_ap_fg=0&ns_ap_dft=0&ns_ap_dbt=0&ns_ap_dit=17673994&ns_ap_as=1&ns_ap_das=0&ns_ap_usage=0&ns_radio=wifi&ns_ap_install=1616780218348&ns_ap_lastrun=1621228815670&ns_ap_ft=0&ns_ap_bt=0&ns_ap_it=17675001&ns_ap_res=1125x2436&ns_ap_sd=1125x2436&ns_ap_po=0x0&ns_ap_lang=en-CH&ns_ap_jb=0&mp_v=3.1.9-beta&mp_brand=RSI
:authority: sb.scorecardresearch.com
user-agent: Play%20RSI/359 CFNetwork/1237 Darwin/20.4.0
accept: */*
accept-language: en-us
accept-encoding: gzip, deflate, br

followed by a page view:

:method: GET
:scheme: https
:path: /p2?c1=19&c2=6036016&ns_ap_an=Play%20RSI&ns_ap_pn=ios&ns_ap_pv=14.5.1&c12=7CC327C611839B7C7B87A938161ED874-cs62&name=play.audio.rete-uno.home&ns_ap_ec=2&ns_ap_ev=view&ns_ap_device=iPhone11%2C2&ns_ap_id=1621246495695&ns_ap_bi=ch.rsi.rsiplayer.beta&ns_ap_pfm=ios&ns_ap_pfv=14.5.1&ns_ap_ver=3.1.9-beta&ns_ap_sv=6.7.1.210208&ns_ap_bv=2.6.1.210205&ns_ap_smv=6.5&ns_type=view&ns_ts=1621246491634&ns_ap_env=0-1-0-0-2-0&ns_ap_ut=60000&ns_ap_ar=arm64&ns_ap_cs=354&ns_ap_fg=0&ns_ap_dft=0&ns_ap_dbt=0&ns_ap_dit=963&ns_ap_as=0&ns_ap_das=0&ns_ap_usage=963&ns_category=play.audio.rete-uno&ns_radio=wifi&ns_ap_ft=0&ns_ap_bt=0&ns_ap_it=963&ns_ap_res=1125x2436&ns_ap_sd=1125x2436&ns_ap_po=0x0&ns_ap_lang=en-CH&srg_n1=play&srg_n2=audio&srg_n3=Rete%20Uno&srg_title=home&srg_ap_push=0&mp_v=3.1.9-beta&mp_brand=RSI
:authority: sb.scorecardresearch.com
user-agent: Play%20RSI/359 CFNetwork/1237 Darwin/20.4.0
accept: */*
accept-language: en-us
accept-encoding: gzip, deflate, br

defagos commented 3 years ago

As discovered by @pyby:

After a phone restart, and if an app was never launched, a received push notification will not wake it up.
If the app has been started and then killed, receiving a push notification will wake it up and go through the -application:didFinishLaunchingWithOptions: delegate method. The launch options dictionary contains something like:

{
    UIApplicationLaunchOptionsRemoteNotificationKey =     {
        "_" = "1440bd16-515a-47f5-bdac-2ff21727dbcf";
        aps =         {
            alert =             {
                body = "Test Media 2 PY iPhone";
                title = "12h45 2 PY iPhone";
            };
            badge = 6;
            "content-available" = 1;
            "mutable-content" = 1;
        };
        "com.urbanairship.metadata" = "eyJ2ZXJzaW9uX2lkIjoxLCJ0aW1lIjoxNjIxMjY4MTA5MTE0LCJwdXNoX2lkIjoiZDgyMTEyMGEtNGNjNi00NTU2LTgyZWYtYzJjOTk3OTk2NjNmIiwicmV0YXJnZXRpbmciOmZhbHNlfQ==";
        imageUrl = "https://www.rts.ch/2019/12/22/13/48/10962113.image/16x9";
        media = "urn:rts:video:10962075";
        show = "urn:rts:show:tv:548307";
        type = newod;
    };
}

defagos commented 3 years ago

I wanted to have a simple way to reproduce this issue in the SRG Analytics demo. I tried with local notifications as well as with background tasks (new BGTaskScheduler from iOS 13), but these do not seem to wake up the application as push notifications can.

defagos commented 3 years ago

Here is a summary of our investigations.

Goal

We were recently reported ghost start events, which Mediapulse or comScore documentation never mention or clearly define, but which have been reported to us as a critical issue.

After discussions with people from Mediapulse, Kantar and comScore, we could understand these events are comScore start events (automatically generated by the comScore SDK once initialized) which occur without user intervention, as a result of the application being summoned in the background for some reason.

In the following we will therefore define ghost starts as comScore start events triggered in the background without any user intervention.

We were asked to eliminate these ghost starts, as they are counted as application startups, but Mediapulse wants be able to count user-initiated startups only.

How to Reproduce

We discovered ghost start events are related to push notifications which might wake up the application in the background. Here is how they can be reliably reproduced:

Setup traffic monitoring, e.g. with Charles proxy.
Open some app, e.g. Play RTS beta and kill it.
Go to Airship web portal and send a push notifiation to Play RTS manually. The ghost start will be logged by the proxy tool.

Step 2 is especially important:

You need to have started the application at least once (after your device was restarted), otherwise push notifications will not be able to wake up the application, and thus no ghost start will be emitted.
You must kill the application because the comScore initialization sequence is made once at application startup.

Investigation

The official comScore integration documentation describes how the comScore SDK must be implemented in applications (paragraph 2.2). The recommendation, which is the same for SRG Analytics, is to start the comScore SDK from the application:didFinishLaunchingWithOptions: method.

This method is called in the following cases:

The user starts the app manually.
A push notification wakes up the application in background (for this to work the application must have been started once and killed). In this second case the launch options parameter contains the payload of the push notifications which triggered the launch.

If you compare the comScore start events emitted in these two scenarios, you can notice there is a label ns_ap_env sent which describes the app environment as a string made of multiple components. The first host state of the application (internally delivered by -[SCORHostApplicationState firstHostState]) can be extracted from the 5th component:

Value 1 means the start was interactive, i.e. the user tapped on the application icon. Example: ns_ap_env = 0-1-0-0-1-0.
Value 2 means the start occurred in background, e.g. through a push notification. Example: ns_ap_env = 0-1-0-0-2-0.

Background View Events

Our investigations revealed that a view event might be emitted in background after application startup, which is likely incorrect. This is not part of the ghost start itself but something we probably want to fix.

Fix Proposal

The comScore SDK already sends everything needed to distinguish interactive starts from ghost starts. We therefore propose to do nothing to mitigate them. We must only ask comScore to filter start events based on the value of ns_ap_env.

This is the best way to proceed:

This is consistent with the comScore SDK integration guide.
This does not require complicated strategies to defer SDK startup if the start was initiated in the background. Application lifecycle can be complicated. The only reliable way to know which state the comScore SDK is in when started is to have the SDK do this work itself and send it as label.
Push notifications are common in media applications. This way we ensure that no media company has to do any implementation work to implement reliable behavior. This is fair and efficient.

The only fix we must do in our SRG Analytics SDK is to prevent view events from being emitted in the background, though. This will be delivered as a patch.

Remark

~If ns_ap_env or no offcial field can be used, I recommend we send the application state at the time the comScore SDK label is started, using a new Mediapulse label. This is reliable, as UIApplicationStateBackground is only seen if the app is truly started in background, not during the time an app is being brought to life following a user interaction.~

This is incorrect. The application might be in the background state in -application:didFinishLaunchingWithOptions: even when started interactively, see official documentation. An additional reason to rely on comScore, which is readily able to correctly determine whether the start was triggered in foreground or in background. We cannot namely reliably augment the start event with the application state.

I made a few tests:

SRG Analytics demo: State is background.
Play SRG and SRF Meteo: State is inactive.

Maybe the reason for this difference is that SRG Analytics demo is a lightweight app, or maybe this is because of some supported capabilities. The exact reason is undetermined, but what I could verify is that the state is not reliable for our purposes at the time the app is started.

defagos commented 3 years ago

I can confirm that SRG Analytics is not used in SRF Meteo widgets, so this is entirely related to the app being woken up as well.

pyby commented 3 years ago

According to Apple documentation, if the push notification payload contains "content-available" : 1, the operating system can launch the application in background to process some update.

https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/pushing_background_updates_to_your_app?preferredLanguage=occ

Discussion: https://developer.apple.com/forums/thread/31296

From AirShip, we can test it:

"content-available" : 1: Wake up the main application in background to process some tasks.
"mutable-content" = 1: Wake up an extension in background to modify notification content (not the application itself).

With AirShip admin UI, when removing this content-available property, the push notification gets the image, but does not launch the application anymore.

As a "hot fix for Play SRG products", I would suggest to remove server side the "content-available" : 1 in the push notification payload.

It won't fix other applications using SRGAnalytics SDK and the discussion should continue to find a solution accepted by Mediapulse, ComScore and a robust implementation in our analytics SDK.

pyby commented 3 years ago

At 27.05.2021, 23:12 GMT+2 🕚 :

The Play SRG middleware for push notifications does not send an iOS payload with "content-available" : 1 anymore (on dev and prod).

defagos commented 3 years ago

Here is a communication to our SDK users. Please read it in its entirety.

🙏 Thanks in advance for your cooperation.

Fixing Mediapulse Analytics Ghost Starts

Mediapulse reported that several SRG SSR applications send what they describe as ghost start events. These are application start events automatically emitted in background by the comScore SDK (the technical solution used for Mediapulse data collection), without the user actually starting the application.

Ghost starts are considered a severe issue by Mediapulse, as they currently cannot be distinguished from interactive application starts, the only ones that Mediapulse is interested in.

The present guide describes what you should do to ensure your application does not emit ghost starts and thus fulfills Mediapulse requirements.

Reason for Ghost Starts

If you have correctly followed our SRG Analytics integration guide your application must initialize our SDK in its application delegate from -application:didFinishLaunchingWithOptions: method implementation. Our SDK in turn initializes the comScore SDK which emits a start event.

The -application:didFinishLaunchingWithOptions: method is called in foreground when the user starts an application interactively. In such cases the start event emitted by comScore occurs in foreground.

If an application receives push notifications with silent background updates, though, the system might silently launch the application in the background, calling the very same delegate method. In this case the comScore SDK will send a start event in background, which is what Mediapulse is asking us to remove.

Applications

The following applications are currently tracked by Mediapulse and might be affected. This list was extracted from an internal wiki page (requires login):

RSI

RSI News
Zerovero
RSI Sport
RSI.ch
Play RSI

RTR

Play RTR

RTS

RTS Info
RTS Sport
Play RTS

SRF

SRF News
SRF Sport
SRF Meteo
Play SRF

SRG SSR

Play Suisse

SWI

SWI Plus
Play SWI

Fixing Ghost Starts

Mediapulse would like to produce audience measurement reports starting with data collected in June 2021. Since June has already begun we were asked for an emergency fix, but introducing a reliable solution at the SRG Analytics SDK level requires coordination with Mediapulse and comScore and is sadly not feasible in the timeframe we were given.

This is why we need your help. Until a proper solution is introduced we need to ask all teams using SRG Analytics to ensure their product is not using silent background updates if not absolutely required.

Please follow one of the procedures below to check your current push notification server configuration and update it appropriately if needed. No application update is required, only your push server configuration might require a minor change.

⚠️ If you need silent background updates to support essential features of your application please get in touch with us so that we can figure out a strategy together.

Application Without Push Notifications

If your application does not use push notifications no change is required.

Push Notifications via Airship

If you send push notifications via Airship please check that your server does not set the content_available flag to true when creating the notification payload sent to Airship.

Remark

You can check the payload of existing push notifications delivered by the Airship API to your application in the Airship web portal, under Messages > Activity Log.

Push Notifications via APNS

If you send push notifications via Apple Push Notification Service (APNS) directly please check that your server does not set the content-available flag to 1 when creating the notification payload sent to APNS.

Reporting

⚠️ Please inform our team when you are done so that we can have a better understanding of the overall situation. This will ensure we can accurately report progress to the SRG SSR GD and Mediapulse.

In particular we would be grateful to know:

[ ] Whether your application uses push notifications or not.
[ ] If your application requires silent background updates.
[ ] If you plan to use silent background updates in the future.
[ ] The Background Modes of your application, which can be seen under the Signing & Capabilities tab of your target in Xcode. This will help us better understand which kinds of behaviors are currently used by SRG SSR applications. A screenshot suffices, see below:

Next Steps

By disabling silent background updates via push notifications, and provided no SRG SSR application uses this feature for vital purposes, all SRG SSR applications should be able to comply with Mediapulse requirements.

In parallel we will discuss a long-term reliable solution for SRG Analytics with Mediapulse and comScore. Once this solution has been accepted it will be delivered as part of our usual SDK updates. Until this solution has been delivered you should not plan features requiring silent background updates.

defagos commented 3 years ago

Status we be updated below as we receive more information

RSI Applications

Application	Status	Push notifications?	Silent background updates now?	Silent background updates planned?	Background modes
RSI News
Zerovero
RSI Sport
RSI.ch
Play RSI	✅	Yes	No	No	Audio / Remote Notifications

RTR Applications

Application	Status	Push notifications?	Silent background updates now?	Silent background updates planned?	Background modes
Play RTR	✅	Yes	No	No	Audio / Remote Notifications

RTS Applications

Application	Status	Push notifications?	Silent background updates now?	Silent background updates planned?	Background modes
RTS Info
RTS Sport
Play RTS	✅	Yes	No	No	Audio / Remote Notifications

SRF Applications

Application	Status	Push notifications?	Silent background updates now?	Silent background updates planned?	Background modes
SRF News
SRF Sport
SRF Meteo
Play SRF	✅	Yes	No	No	Audio / Remote Notifications

SRG SSR Applications

Application	Status	Push notifications?	Silent background updates now?	Silent background updates planned?	Background modes
Play Suisse	✅	No	No	No

SWI Applications

Application	Status	Push notifications?	Silent background updates now?	Silent background updates planned?	Background modes
SWI Plus	✅	Yes	No	No	Remote Notifications / Background Fetch
Play SWI	✅	Yes	No	No	Audio / Remote Notifications

defagos commented 3 years ago

Reasons for which the application might be launched indirectly are provided by launch options.

A possible temporary workaround for applications which might not be able to remove silent background updates, or whose ghost starts might be triggered for another reason, could be to check whether a launch option exists. For most options, except for shortcut items and user activity continuation, SRGAnalyticsTracker should probably not be started. The documentation even states that in general this dictionary will be empty if the user started the application directly, which is what we want to detect.

This of course will prevent the app from being tracked for the session if the user opens the app right afterwards. But since we gain no competitive advantage as we are removing events, I think this approach could be acceptable.

SebastienChauvin commented 3 years ago

Play Suisse: no push notification. So, according to your doc @defagos, we should be all good, is that correct ?

defagos commented 3 years ago

Play Suisse: no push notification. So, according to your doc @defagos, we should be all good, is that correct ?

Yes, I think Play Suisse should not suffer from ghost start issues if you don't have any push notifications at all.

defagos commented 3 years ago

We had a meeting with Mediapulse, Kantar and the GD to assess the situation. We could together define what needs to be measured and what ghost starts exactly are.

What we want to measure are user interactions, i.e. any application event occurring as result of interacting with the app directly, in foreground.

Ghost starts (resp. views) were defined as ny application start (resp. view) event occurring without user interaction, in the background.

Mediapulse uses starts as a sign of user interaction, this is why ghost starts are an issue. Ghost views could have been an issue for SRG Analytics but we prevented them entirely in our latest release.

The slides of the presentation are available.

defagos commented 3 years ago

Following our last meeting we promised to deliver a sample project to reproduce the ghost start in isolation, without any of the SRG SSR stuff we have.

The resulting project is available on a separate repository with an in-depth analysis of the issue.

Here is an excerpt of the mail we sent to Mediapukse, Kantar and the GD to inform them of our findings:

As promised we worked on a sample project to reproduce ghost starts in a simple setup. Our sample project proves that ghost starts are not related to SRG SSR app specifics, but are potentially affecting all apps from Swiss media publishers involved in the Mediapulse initiative.

As already conjectured it suffices that an application integrates comScore in the recommended way to open the door to ghost starts, provided the operating system is able to wake up the application in the background (silent push notification, location updates, etc.)

We also found the reason why this issue was discovered in SRG SSR apps but was seemingly not affecting apps from other publishers (which is simply not true in general). We namely widely use Airship as push notification solution and discovered that the combination of the Airship and comScore SDKs in the same project suffices to alter the way comScore behaves when the application runs in the background, making ghost starts easier to detect. In presence of Airship ghost starts are namely immediately emitted by the comScore SDK when the system wakes up the app, while this process is deferred to the next interactive app launch when Airship is not used.

Our sample project provides a way to verify all these claims, with videos showing what happens in each scenario.

Given the results we found we are now confident there is nothing more we can do at the SRG SSR level to comply with Mediapulse guidelines regarding ghost starts. The solution is simply not within our hands, especially since we have no access to comScore source code for further investigation.

The results of our in-depth analysis suggest that:

Mediapulse should probably reconsider how user interactions are counted, maybe by looking at page views only, ignoring start events entirely. Given the fact we are already supposed to send page views when the user launches an application or resumes it to the foreground this is appears to be the simplest solution.

If the comScore SDK uses start timestamps as a reference for calculating usage times, ghost starts and the way Airship interferes with them might require further technical investigation. If this creates a positive bias in the calculated durations this issue could potentially affect other comScore implementations around the world as well, but as we don’t know how the comScore SDK works internally this might just be a pessimistic assumption.

TL;DR Ghost starts are not specific to SRG SSR but a general issue. We recommend that user interactions are measured solely using page views, ignoring start events. Moreover we found a behavioral conflict with Airship, whose criticality level needs to evaluated.

defagos commented 3 years ago

The issue was discussed with Mediapulse and comScore. It was recognized that the ghost start issue is a byproduct of the solution and that this has to be fixed on their side. They will filter them on their side, apps don't have to do anything in particular to suppress them.

A final communication was sent to our library users to inform them nothing has to be fixed on their side. All that is required is to use SRG Analytics 7.0.4, ensure the tracker is started from the app entry point as documented.

If some apps introduced temporary workarounds (e.g. disabling silent push notifications) and this disabled non-essential features, those workarounds can now be removed.

I consider this issue closed.

SRGSSR / srganalytics-apple

comScore ghost starts #54

How to reproduce the issue

App lifecycle

Fix

Goal

How to Reproduce

Investigation

Background View Events

Fix Proposal

Remark

Fixing Mediapulse Analytics Ghost Starts

Reason for Ghost Starts

Applications

RSI

RTR

RTS

SRF

SRG SSR

SWI

Fixing Ghost Starts

Application Without Push Notifications

Push Notifications via Airship

Remark

Push Notifications via APNS

Reporting

Next Steps

RSI Applications

RTR Applications

RTS Applications

SRF Applications

SRG SSR Applications

SWI Applications