Write QoS specifications

[defagos]

As a Pillarbox team member I need to better understand how Pillarbox behaves in production.

Acceptance criteria

• We have a common understanding of how we can use Snitch.
• The data format we will use for QoS is documented.
• Dedicated implementation stories are ready.

Tasks

• [x] Discuss the matter within the team.
• [x] Document format.
• [x] Discuss understanding of what a session is (Android: Replaying the current item is made in the same session, different items belon to different sessions, though).
• [x] Discuss if QoS can be enabled and disabled on-the-fly client-side, in a way similar to analytics trackers.
• [x] Is QoS related to SRG SSR content only or will we also track performance of all other sources?
• [ ] GDPR compliance?

[defagos]

Excellent WWDC session about KPIs we can extract and how. Here are the associated slides for quick reference. Associated sample code and video can also be found online.

[defagos]

First draft:

[jboix]

See here the format for a START event payload:

{
  "session_id": "1",
  "event_name": "START",
  "data": {
    "device_id": "MAC / ISBN",
    "device_model": "Samsung Galaxy S24 / iPhone15,7",
    "device_type": "TV / Car / Phone / Tablet",
    "screen_width": 1280,
    "screen_height": 720,
    "os_name": "android / iOS / macOS / windows / linux ",
    "os_version": "11.23",
    "browser": "safari / chrome / firefox",
    "browser_version": "11.0",
    "player_name": "pillarbox / video.js / letterbox",
    "player_platform": "android / apple / web",
    "player_version": "2.0.1",
    "origin": "ch.srgssr.srf-meteo / www.rts.ch/info/article/1234",
    "media_id": "urn:rts:video:1234",
    "media_source": "https://il.srgssr.ch/composition/?urn=urn:rts:video:1234",
    "time_metrics": {
      "media_source": 1200,
      "asset": 1500,
      "drm": 100,
      "total": 2856
    },
    "timestamp": 1717665997932
  }
}

And an example of how it can be stored in the database:

device_id	device_model	...	payload
iOS	iPhone15.7	...	{"session_id": "1", "event_name": "START", "data": {"device_id": "MAC / ISBN" ... }}

Remarks:

• For the web the browser version is not available yet. The format above represents an ideal result but some fields might not be provided yet (the web team does not want to pull additional dependencies just for the sake of parsing the user agent).
• The START event can only be sent after the timings have been recorded, but its timeestamp must be the time at which the item being played becomes active.
• Timings / information that are missing must not be provided in the JSON.

[defagos]

See here the format for an ERROR event payload:

{
    "event_name": "ERROR",
    "session_id": "1a2b3c-xyz",
    "timestamp": 1717665997932,
    "data": {
        "severity": "WARNING / FATAL",
        "name": "ERR_001",
        "message": "Not found",
        "player_position": 11000,
        "log": "Raw error information as returned by the player, for example stack trace"
    }
}

Some remarks:

• player_position in milliseconds.
• If the player_position cannot be extracted (e.g. error before playback even started) then omit.
• An ERROR is always preceded by a START. The START only provides timings that are available (e.g. no drm timing is provided if the media source could not even be loaded).

[defagos]

See here the format for an event payload:

{
    "event_name": "BUFFER / HEARTBEAT / STOP / STALL / DISCONTINUITY / ...",
    "session_id": "1a2b3c-xyz",
    "timestamp": 1717665997932,
    "data": {
        "url": "https://www.rts-vod.akamai.com/content/segm10.ts",
        "bitrate": 1000000,
        "bandwidth": 2000000,
        "buffer_duration": 120000,
        "stall_count": 2,
        "stall_duration": 1000,
        "playback_duration": 40000,
        "player_position": 10000
    }
}

Some remarks:

• Events above are loosely defined for the moment. What is important is that the model can be easily extended to support new events. We might start with only a few events (e.g. heartbeat, stop) and see if other events (discontinuity, stall) can be useful later.
• bitrate is the quality of the variant currently being played, in bytes per second.
• bandwidth is the device-measured network bandwidth, in bytes per second.
• buffer_duration is the forward buffered duration, measured in milliseconds.
• stall_count is the number of times the player buffers during playback, not as a result of a seek.
• stall_duration is the corresponding total duration, in milliseconds.
• playback_duration and player_position in milliseconds as well.
• Unlike analytics we have a single STOP for normal playback end and playback interruption (e.g. tab or player view closed by the user) since the distinction is not really relevant for QoS / QoE.

Open question

Cumulative values can be sent in different ways to create charts that show total or increments:

• Total values only. In this case the graph tool we plug to the data must be able to calculate the differences between totals to display increments.
• Increments only. In this case the graph tool we plug to the data must be able to calculate the totals from the addition of the increments. We might also encounter more issues if for some reason an increment is missed.
• Totals and increments. In this case all the data is available for the graph tool but of course each client must compute and send both.

We should investigate what is possible with graph tools before we decide what we send.

[defagos]

TL;DR We keep things as simple as possible and avoid introducing complex logic where no strict alignment is required.

Playback sessions

• On Android and Apple platforms a player session ID is associated with an item. If we implement repeat mode then we need two items and thus we will have two session IDs as a result. If we have a single item and pause at item end we stay in the same session if we seek back, though.
• On the web repeat mode replays the same item and thus the session stays the same.

In the end, and from a QoS perspective, having the same media counted as two sessions or seen as a single one should not be relevant. Therefore we should keep each platform implementation as simple as possible and identify sessions in the simplest way on each.

Enabling / disabling QoS on the fly?

• For SRG SSR URN-based content we must not be able to disable QoS. It is mandatory.
  • On Apple platforms this means we must have a way to force a tracker to be always enabled, even when isTrackingEnabled has been set to false.
  • For the web some investigation is required.
  • If several players play at the same time all of them will send QoS data.

Is it a problem if we send QoS data about 3rd party content to our QoS service?

• On Apple platforms nothing special is required, no matter what we want to achieve. If some content requires QoS, inside or outside the company, we can attach a dedicated tracker sending data where needed (flexible URL configuration per tracker).
• On Android the URL to send QoS data to is per-player instance. So the same player will send data from any content being played (inside our outside SRG SSR) to the chosen URL. It shouldn't be an issue if our QoS also collects data about other content being played in such cases.
• On the web QoS will probably be set per content being played, similarly to Apple platforms.

GDPR compliance

As long as we don't store any IP address server-side, the data we send should be safe. But we should check with the data officer first.