Write QoS specifications

[defagos]

As a Pillarbox team member I need to better understand how Pillarbox behaves in production.

Acceptance criteria

• We have a common understanding of how we can use Snitch.
• The data format we will use for QoS is documented.
• Dedicated implementation stories are ready.

Tasks

• [x] Discuss the matter within the team.
• [x] Document format.
• [x] Discuss understanding of what a session is (Android: Replaying the current item is made in the same session, different items belon to different sessions, though).
• [x] Discuss if QoS can be enabled and disabled on-the-fly client-side, in a way similar to analytics trackers.
• [x] Is QoS related to SRG SSR content only or will we also track performance of all other sources?
• [x] GDPR compliance?

[defagos]

Excellent WWDC session about KPIs we can extract and how. Here are the associated slides for quick reference. Associated sample code and video can also be found online.

[defagos]

First draft:

[jboix]

See here the format for a START event payload:

{
    "session_id": "1",
    "event_name": "START",
    "timestamp": 1717665997932,
    "data": {
        "time_metrics": {
            "metadata": 1200,
            "asset": 1500,
            "token": 1000,
            "drm": 100,
            "total": 2856
        },
        "screen": {
            "width": 1280,
            "height": 720
        },
        "os": {
            "name": "android / iOS / macOS / windows / linux ",
            "version": "11.23"
        },
        "player": {
            "name": "pillarbox / video.js / letterbox",
            "platform": "android / apple / web",
            "version": "2.0.1"
        },
        "browser": {
            "name": "safari / chrome / firefox",
            "version": "11.0"
        },
        "media": {
            "id": "urn:rts:video:1234",
            "metadata_url": "https://il.srgssr.ch/composition/?urn=urn:rts:video:1234",
            "asset_url": "https://akamai.com/quality_content.m3u8",
            "origin": "ch.srgssr.srf-meteo / www.rts.ch/info/article/1234"
        },
        "device": {
            "id": "MAC / IDFV",
            "model": "Samsung Galaxy S24 / iPhone15,7",
            "type": "TV / Car / Phone / Tablet / Desktop / Headset"
        }
    }
}

And an example of how it can be stored in the database:

device_id	device_model	...	payload
iOS	iPhone15.7	...	{ JSON }

Remarks:

• For the web the browser version is not available yet. The format above represents an ideal result but some fields might not be provided yet (the web team does not want to pull additional dependencies just for the sake of parsing the user agent).
• The START event can only be sent after the timings have been recorded, but its timestamp must be the time at which the item being played becomes active.
• Timings / information that are missing must not be provided in the JSON.
• metadata and asset represent QoE values, i.e. they represent the value perceived by the user (0 if preloading was successful).

[defagos]

See here the format for an ERROR event payload:

{
    "event_name": "ERROR",
    "session_id": "1a2b3c-xyz",
    "timestamp": 1717665997932,
    "data": {
        "severity": "WARNING / FATAL",
        "name": "ERR_001",
        "message": "Not found",
        "player_position": 11000,
        "url": "https://www.rts-vod.akamai.com/content/segm10.ts",
        "log": "Raw error information as returned by the player, for example stack trace"
    }
}

Some remarks:

• If the player_position cannot be extracted (e.g. error before playback even started) then omit.
• An ERROR is always preceded by a START. The START only provides timings that are available (e.g. no drm timing is provided if the media source could not even be loaded).

[defagos]

See here the format for an event payload:

{
    "event_name": "BUFFER / HEARTBEAT / STOP / STALL / DISCONTINUITY / ...",
    "session_id": "1a2b3c-xyz",
    "timestamp": 1717665997932,
    "data": {
        "url": "https://www.rts-vod.akamai.com/content/segm10.ts",
        "bitrate": 1000000,
        "bandwidth": 2000000,
        "buffer_duration": 120000,
        "stall": {
            "count": 2,
            "duration": 1000
        },
        "playback_duration": 40000,
        "player_position": 10000
    }
}

Some remarks:

• Events above are loosely defined for the moment. What is important is that the model can be easily extended to support new events. We might start with only a few events (e.g. heartbeat, stop) and see if other events (discontinuity, stall) can be useful later.
• bitrate is the quality of the variant currently being played, in bytes per second.
• bandwidth is the device-measured network bandwidth, in bytes per second.
• buffer_duration is the forward buffered duration, measured in milliseconds.
• stall.count is the number of times the player unexpectedly buffers during playback, not as a result of a seek.
• stall.duration is the corresponding total duration, in milliseconds.
• playback_duration in milliseconds as well. playback_duration is measured in wall-clock time, independent of playback speed settings (i.e. playback_duration = 60000 if a user watches 60 seconds of content, no matter the playback speed).
• player_position in milliseconds. For DVR streams the offset from the live edge (collapsing to 0 if there is no window; the value is never omitted).
• Unlike analytics we have a single STOP for normal playback end and playback interruption (e.g. tab or player view closed by the user) since the distinction is not really relevant for QoS / QoE.

Open question

Cumulative values can be sent in different ways to create charts that show total or increments:

• Total values only. In this case the graph tool we plug to the data must be able to calculate the differences between totals to display increments.
• Increments only. In this case the graph tool we plug to the data must be able to calculate the totals from the addition of the increments. We might also encounter more issues if for some reason an increment is missed.
• Totals and increments. In this case all the data is available for the graph tool but of course each client must compute and send both.

We should investigate what is possible with graph tools before we decide what we send.

[defagos]

TL;DR We keep things as simple as possible and avoid introducing complex logic where no strict alignment is required.

Playback sessions

• On Android and Apple platforms a player session ID is associated with an item. If we implement repeat mode then we need two items and thus we will have two session IDs as a result. If we have a single item and pause at item end we stay in the same session if we seek back, though.
• On the web repeat mode replays the same item and thus the session stays the same.

In the end, and from a QoS perspective, having the same media counted as two sessions or seen as a single one should not be relevant. Therefore we should keep each platform implementation as simple as possible and identify sessions in the simplest way on each.

Enabling / disabling QoS on the fly?

• For SRG SSR URN-based content we must not be able to disable QoS. It is mandatory.
  • On Apple platforms this means we must have a way to force a tracker to be always enabled, even when isTrackingEnabled has been set to false.
  • For the web some investigation is required.
  • If several players play at the same time all of them will send QoS data.

Is it a problem if we send QoS data about 3rd party content to our QoS service?

• On Apple platforms nothing special is required, no matter what we want to achieve. If some content requires QoS, inside or outside the company, we can attach a dedicated tracker sending data where needed (flexible URL configuration per tracker).
• On Android the URL to send QoS data to is per-player instance. So the same player will send data from any content being played (inside our outside SRG SSR) to the chosen URL. It shouldn't be an issue if our QoS also collects data about other content being played in such cases.
• On the web QoS will probably be set per content being played, similarly to Apple platforms.

GDPR compliance

As long as we don't store any IP address server-side, the data we send should be safe. But we should check with the data officer first.

[defagos]

@amtins contacted the RTS team but got no answer. Maybe we should ask the GD data governance team directly.

[defagos]

The mail has been sent. Moving this task to Follow while waiting for an answer.

[amtins]

Summary of the meeting with the legal department of the GD

1. The declaration to User Centrics is mandatory to ensure transparency with our users. When QoS is ready to go live, we'll need to contact him again to draft the text for User Centrics to distribute to all business units. Counterparts in the various business units will also be copied on the e-mail.
2. The Pillarbox QoS service will have a necessary status and cannot be deactivated by the user. I have stressed that the purpose of this service is to ensure that what needs to work works, so that our users can enjoy the best possible experience, and not to feed traffic analysis tools.
3. The collection of the user's IP address is authorized for Pillarbox QoS. I've highlighted the case of users who were geolocated abroad even though they were in Switzerland.