Definition around use of CMCD for beacons enabling out-of-band QOE Reporting

[glenng1215gh]

While CMCD version 1 defines a JSON format for the payload, it does not fully address the use case where the primary goal is to report player-side QOE data out-of-band to an analytics service, one that is typically independent from any of the CDNs. In this use case, player QOE data tends to be session and time-interval based as opposed to based around individual media segment requests.

With a few additions, CMCD can essentially also become a player QOE beaconing standard. A beacon would be defined as a set event-oriented beacon types, with each beacon type containing a set of CMCD keys. A proposed set of beacon types could like something like this:

playback-start Issued at play start with all of the CMCD-Session keys and selected CMCD-Object keys such as Top bitrate (tb)

playback-end Issued at play end with the CMCD sessionID (sid) key

playback-error Issued whenever the player encounters a playback error. payload would carry an error identifier & data (TBD)

playback-stall Issued whenever the player encounters a buffer stall. contains CMCD keys such as br,mtp,ot,pr,rtp. It would also be useful to capture duration of the stall, either on this event (posted when the stall ends), or with an additional playback-resume event.

playback-heartbeat Issued on a time interval defined either by the player or the analytics service that is is reporting, with a typical interval around 30 seconds. Metrics such as bitrate, throughput, and buffer length would represent the averages over the reporting time interval. Contains CMCD keys such as br,bl,mtp,ot,pr,rtp

Some may argue that this is beyond the scope of CMCD, but it could bring value to the overall ecosystem. For example, the existence of a QOE beaconing standard could eliminate the need for QOE vendor-specific player plugins, with players and QOE analytics backends all understanding the same beacon specification.

[PCaponetti]

The notion of batch reporting came up in the 2023-11-02 meeting that potentially overlaps here.

[glenng1215gh]

Additional events that could be captured with these beacons:

• Rendition switches
• Player-initiated mid-stream CDN switches

[acbegen]

What is the relation of this to CTA-2066? There are companies who have been using 2066 definitions and if we introduce new definitions here, would not that cause more confusion/interop issues?

[glenng1215gh]

I was unaware of CTA-2066. good work there streaming QOS events and metrics. What I don't see there is a definition of a transmission protocol or format for these events (JSON or XML posts over HTTPS, for example). Now that I see CTA-2066, that should certainly form the basis of a QOS beaconing standard, and should be harmonized with CMCD to use common names and definitions whenever possible. Is any new work ongoing on CTA-2066?

[acbegen]

Yup, 2066 did not bother with the transport side of the metrics but then when we defined CMCD, we did not really tie it to 2066, either. Coming from the same organization, these two specs could have been used together in a better way. I think this warrants some discussion in the next call. 2066 was not a product of the WAVE project but it is from R04 WG20 Streaming Media Quality of Experience (QoE). @mlevine84 and @wilaw might say more. Another good read from Akamai is here.

[sebastian-siepe]

Hi @glenng1215gh , hi @acbegen ,

we also see a huge benefit in extending CMCD to a full beaconing standard, which provides detailed QoS/QoE metrics.

Benefits are:

• Elimination of player side library integrations
• Unified standard across different players/platforms/environments
• Comparability of QoS/QoE metrics across different player/platforms/environments

In order to better understand which metrics are needed for a full picture about the QoS/QoE for each streaming session, we created a table (see below) which includes:

• All metrics we think are relevant for a full beaconing standard
• Overview if metrics are already covered by CMCD v1
• Overview it metrics are already covered by CTA-2066
• Definition and further requirements for CMCD v2

One note about CTA-2066: We have the feeling that CTA-2066 is a good description on how to track metrics of a media session in general, but it does not define a very detailed set of metrics.

Happy to discuss the general approach in our next call!

Value	Type	When to Transmit	CMCD v1	CTA-2066	Definition & Further Requirements for CMCD v2
Playback Session definition	behaviour		yes		Definition of playback session
Add-insertion definition	behaviour		yes		Definition of add-insertion
Session Start	event	onEvent	partially (first request)	partially (playbackRequest)	Event when session start is requested; event time in milliseconds; Related issue
Session End	event	onEvent		partially (playbackFinish)	Event when session is ended; event time in milliseconds; playhead position in milliseconds; additional beaconing request needed, as for VoD no further media segment requests are made
Ad Break Start	event	onEvent		partially (adBreakStart)	Event when ad break starts; event time in milliseconds; playhead position in milliseconds
Ad Break End	event	onEvent		partially (adBreakEnd)	Event when ad break ends; event time in milliseconds; playhead position in milliseconds
PlayState: Connecting	event	onEvent	partially (first request)		Event when playstate Connecting occurs; event time in milliseconds; playhead position in milliseconds; same as session start
PlayState: Playing	event	onEvent		partially (playbackStart)	Event when playing playstate occurs; event time in milliseconds; playhead position in milliseconds
PlayState: Pause	event	onEvent		partially (playbackPause)	Event when pausing playstate occurs; event time in milliseconds; playhead position in milliseconds; additional beaconing request needed, as long pauses can lead to no further data being loaded and a keep alive is necessary to understand the session is still running
PlayState: Buffering/Stall	event	onEvent	partially (bs)	partially (playbackStall)	Event when stalling playstate occurs; exact timing in milliseconds; playhead position in milliseconds
PlayState: Seek Start	event	onEvent		partially (seekStart)	Event when seeking start playstate occurs; exact timing in milliseconds; playhead position in milliseconds
PlayState: Seek End	event	onEvent		partially (seekEnd)	Event when seeking end playstate occurs; exact timing in milliseconds; playhead position in milliseconds
PlayState: Warning	event	onEvent			Event when warning playstate occurs; playstate definition; exact timing in milliseconds; playhead position in milliseconds; need to include error type, error message, error description metrics; Related issue
PlayState: Error	event	onEvent			Event when error playstate occurs; playstate definition; exact timing in milliseconds; playhead position in milliseconds; need to include error type, error message, error description metrics; Related issue
PlayState: End	event	onEvent		partially (playbackFinish)	Event when ended playstate occurs; event time in milliseconds; playhead position in milliseconds; additional beaconing request needed, as for VoD no further media segment requests are made
CDN Switch	event	onEvent			Event when CDN switch event occurs (first request to new CDN); event time in milliseconds; playhead position in milliseconds; CDN decision: default, error fallback multi-CDN decision (e.g., content steering server response)
User Interaction	event	onEvent			Event when user interaction occurs; user interaction with video player, such as "requested to play", "requested to pause", "requested new video bitrate", "requests captions", etc.
Rendition Update	event	onEvent	partially (renditionUpdate)		Event when rendition update occurs; exact timing in milliseconds; playhead position in milliseconds; rendition width; rendition height
Player Resize	event	onEvent	partially (playerResize)		Event when a player resize occurs; exact timing in milliseconds; playhead position in milliseconds
Playback Failure	event	onEvent	partially (playbackFail)		Event when a playback failure occurs, indicating that playback failed ultimately; exact timing in milliseconds; playhead position in milliseconds; Related issue
Session ID	metric	each request (mandatory)	yes (sid)		As defined in CMCD v1; needs to be mandatory
Relative Time since Playback Start	metric	each request (mandatory)			Relative time in milliseconds since session start
Sequential Beacon Number	metric	each request (mandatory)			Human-readable error description. Providing further error information, such as a stack trace, specific platform information, etc. Could be included in Play State Error
Playhead Position	metric	each request (mandatory)	yes		Current playhead position of media player in milliseconds
Bitrate	metric	each request (mandatory)	yes (br)	yes (videoReportedBitrate & audioReportedBitrate)	As defined in CMCD v1
Session State	metric	each request (mandatory)			Current state of the session, such as "Starting", "Playing", "Paused", "Stopped", "Finished". Could also be derived from playStates
Buffer Length	metric	each request (optional)	yes (bl)		As defined in CMCD v1
Throughput	metric	each request (optional)	yes (mtp)		As defined in CMCD v1
Round Trip Time	metric	each request (optional)			Network round trip time of the last request; duration in milliseconds from request start to first byte for the last segment
Object Type	metric	each request (optional)	yes (ot)		As defined in CMCD v1
Requested Maximum Throughput	metric	each request (optional)	yes (rtp)		As defined in CMCD v1
Streaming Format	metric	each request (optional)	yes (sf)		As defined in CMCD v1
Stream Type	metric	each request (optional)	yes (st)		As defined in CMCD v1
Startup	metric	each request (optional)	yes (su)		As defined in CMCD v1
Top Bitrate	metric	each request (optional)	yes (tb)		As defined in CMCD v1
Playback Rate	metric	each request (optional)	yes (pr)	yes (playbackRate)	As defined in CMCD v1
Total Stall Time	metric	each request (optional)			Total amount of stall time in milliseconds since session start
Time to Live Edge	metric	each request (optional)			Definition of delay to server live edge in milliseconds; Only applicable for live streams, especially interesting for low-latency content
Device Type	metric	onEvent: sessionStart, cdnSwitch			Definition of device type, such as "smartphone", "television", "set-top-box", etc.
Device Model	metric	onEvent: sessionStart, cdnSwitch			Definition of model name (if it can be derived from the platform), such as "SM-X710", etc.
Operating System	metric	onEvent: sessionStart, cdnSwitch			Definition of operating system such as "iOS", "Windows", "Android", etc.
Operating System Version	metric	onEvent: sessionStart, cdnSwitch			Definition of operating system version
DRM	metric	onEvent: sessionStart, cdnSwitch			Definition of any kind of DRM information, exact data format TBD
Referrer/App Name	metric	onEvent: sessionStart, cdnSwitch			Definition of Referrer / website or application name
App Version	metric	onEvent: sessionStart, cdnSwitch			Definition of version of website or application
Player Name	metric	onEvent: sessionStart, cdnSwitch			Definition of media player, such as "ExoPlayer", "Shaka", "dash.js", etc.
Player Version	metric	onEvent: sessionStart, cdnSwitch			Definition of media player version number
Content ID	metric	onEvent: sessionStart, cdnSwitch	yes (cid)	yes (contentId)	As defined in CMCD v1
Error Message	metric	onEvent: playstate error, playstate warning			Definition of human-readable error message
Error Type	metric	onEvent: playstate error, playstate warning			Definition of error type, such as "network", "media", "player", "delivery", etc.
Error Description	metric	onEvent: playstate error, playstate warning			Definition of human-readable error description
Custom Data	metric	onEvent: next request			Definition of custom data format, e.g., for A/B testing; Related issue; A/B Testing
Captions Enabled	metric	onEvent: next request			Definition if captions are rendered and which ones are being displayed if multiple are available
Mute/Unmute	metric	onEvent: next request			Flag indicating if playback is muted
Picture in Picture	metric	onEvent: next request			Flag indicating if the player is in picture-in-picture mode
Period/Add Information	metric	onEvent: addBreakStart, addBreakEnd			Information about the current period/add which is being played, TBD

[acbegen]

Wow, this is pretty detailed. Maybe, our first step should be to acknowledge whether there is a need for this (or not). Depending on that, reviving/revising 2066 might be one of the next steps.

[nicolaslevy]

I completely agree with @acbegen on the need to determine if this expansion of CMCD is the way we want to go. My perception is that there's definitely a lot of enthusiasm for evolving CMCD into a standard with more capabilities for QoE reporting.

However, if we decide to go down this path, I think it's important to think about CMCDv2's scope and ensure it doesn't become overly complex for implementation. We wouldn't want a standard that's so intricate that it ends up not being adopted. Planning for incremental enhancements, adding use cases (more information) in future versions as the industry adopts CMCDv2's new functionalities, could be a good approach.

[glenng1215gh]

to minimize overloading CMCD, we could either evolve CTA2066 into a beacon standard (with transmission semantics), or develop a new standard specific only to beacons, leveraging CTA2066 event & metric definitions. either way, CTA2066 and CMCD2 need to be harmonized, using common vocabulary whenever possible.

[PCaponetti]

Notes from meeting 11/30/23

Glenn: QoE used to be a beacon based concept with heartbeats (30 seonds?) where the experience was described. It would be good to have a standard that all could use for this type of use case. Sebastian: +1 for extending CMCD for more QoS/QoE. Looking at CTA-2066, wondering what the overlap is. It doesn't look sufficient. Nicholas: Worried that making CMCDv2 too heavyweight could cost us in adoption. Glenn: CMCDv2 shouldn't be extended to be a beaconing system, but CMCDv2 could be a way to send a common set of information in a standardized QoE "suite". Possibly session based stats should be outside of CMCD which is optimized for request level. Sebastian: Splitting this up into multiple standards and multiple versions will cause a lot of latency before we get valuable QoE data. PJ: +1 to Glenn and Nicholas where we don't want to make this too big. Might go past the original mission of CMCD Will: I was one of the authors of 2066, along with Mux and Conviva. Nobody has used the standard for anything, including Mux developed their own way of computing metrics. CMCD shouldn't dabble in aggregations, but supply metrics/data. CMCD is getting a ton of adoption, and further adoption is paramount. Piers: (also author of 2066) Certain subsets of QoE data can and should be added to CMCD, cited other issues with proposals for specific metrics. If we can get sufficient information for CMCD as-is maybe we don't need to worry about extending it to be a beacon. Chris L: If we do a good job and define all the values, the question of using the JSON format for beaconing is optional and potentially useful. Worried about where the beaconing data is going and potentially re-inventing third party cookies. David H: Greatest value is our adoption, move beaconing to CMCDv3 Ali: Will, what do you think Apple needs in v2? Will: Not sure. Don't even know if they like v1. Paul: We can do a lot with a little, don't need to go as far as aggregations and dashboards but coming up with some pointed and opinionated standards for calculating the most important parts of QoE and be a data provider for exactly that. Also encompassing the failure mode of not having connectivity to the original video provider would be nice. Glenn: Maybe it makes sense to have a fallback if you can't get to the original video provider Alex G: Agree that we can move beaconing into 2066v2, but +1 for quality metrics being in CMCDv2 Piers: See the value in beaconing to reduce reporting load Nicholas L: Might make sense to define some structures Will: hearing a movement towards not defining a beaconing standard, but adding some QoE metrics to CMCDv2 and possibly to allow for the case where the player fails to connect. Would it be useful for CMCD to define a way to report data that is decoupled from a media request? Alex G: +1 to have a way to report that is decoupled from the media requests themselves. Sebastian: +1 to what Paul said earlier, sending essential metrics to understand what is going on. Paul: will Mux or Conviva ever use CMCD as their API? Will: likely no, also we don't want to hold that burden Paul: so we should always have customers looking at their proprietary Mux's and Conviva's, we should define the audience we are targetting, maybe the CDN for decisioning? Will: we should aim to get 90% of the way there, and get the most value out of the least data. Sebastian: +1 to not starting a race with analytics providers, but if we extend in a little way we could get a lot of data. Alex G: Main benefit of CMCD is that it doesn't lock you into a particular solution. Paul: Maybe there is a use case for aggregated QoE across workloads? Will: Edgeio provides global aggregate CMCD data for free

There isn't enough support for full beaconing, there may be support for sending data to external locations that are not the CDN.

[wilaw]

As a summary of the decisions reached on this issue:

1. We'll not attempt to make CMCD v2 a stand-alone replacement for analytics vendors data collection. This goal would bloat the key dictionary, has little chance of completely succeeding and would threaten the core attribute of CMCD v1 which is simplicity - of both comprehension and implementation.
2. We will investigate being able to generate a sub-set of interval-based (rather than request-based) metrics which can be reported independently of the media requests. This concept is captured by issue #117.
3. We will add keys to enabled improved QoE metrics for the request-based reporting.