cta-standards / R4WG20-QoE-Metrics

Issue tracking repository for the R4-Wg20 QoE Initiative
9 stars 2 forks source link

Comments made by John Simmons #21

Closed mlevine84 closed 5 years ago

mlevine84 commented 5 years ago
  1. Apart from the patent disclosure, I only see one must/shall in the spec. “For structured properties it shall also be possible for the analytics client to directly request the value of a single sub-property.” Are their no other normative aspects to this spec?

  2. Lots of references in definitions which are not listed as normative references. E.g. ITU-T Recommendation G.1022, IETF RFC7799, etc. Are these informative/normative/unneeded?

  3. Definitions section begin with two paragraphs, “A data structure…” and “Drawing from ITU-T…” – but there is no term in the term column. ???

  4. Media Time definition – hard to understand from this reading how this is done for a linear stream. Later on when the concept of Playback Sessions are introduced, it starts to make sense, but I would expect the term “Playback Session” to be used in the definition of Media Time.

  5. Playhead Position … “what is a ‘visual cue’??

  6. Playback Session uses the term Primary Content which is undefined.

  7. QoE (Quality of Experience) “The degree of delight or annoyance of the user of an application or service” – should be “to the user”, right?

  8. The Term “Aggregation Set” is defined and never used. If the intent is that “Set” be shorthand for Aggregation Set, it should be stated in the definition.

  9. The PP “streaming media is video and/or audio content …” seems to have nothing to do with “Aggregation Set” ???

  10. Media Player is defined but then used in the text lower case – i.e. as if not a defined term. 12. Symbols and Abbreviations – no abbreviations given???

  11. Introduction redefines Media Player. Unnecessary

  12. Figure 1 – no reason to include anything in this figure that isn’t specified. For example, “operating system” – though I believe most real media players will get some of the properties associated with events from a component that is not strictly speaking the “media player”. For example, if the prototypical media player is a JavaScript app using a browsers controls, and it is using MSE, then some of the events are simply unknown to the media player (might be the need for enhancements to browser event model – as has been discussed in this context).

  13. “Player Buffer Length”, “Buffer State Change”, “Playing State Change” used in the description of an Analytics Client but never defined or used.

  14. Definition of a Set has the sentence “In the scheme of metrics proposed under this standard, “the context” is defined by the population of Playback sessions that play party to measurement, hereafter referred to as the “SET”. What does “play party” mean?

  15. Also in “Definition of a Set” - The exact constructs of a SET are not defined as part of the standard, and are left to Vendors/Service Providers. It is a goal of this standard to provide a measurement framework for relatable comparisons across vendor implementations, while leaving room for independent innovations that may drive vendor specific differentiation. This information belongs in the “Scope” section – but then you need to define “attribute access”.

  16. Definition of Set - PP2 includes a definition of Context. Should be a defined term.

  17. I have concerns about defined terms – e.g. playbackFailed is true if any point during the Playback Session the playbackFail event occurred. But what is the definition of playbackfailed? If it pauses for 30 seconds and then restarts, is that a fail? A minute? An Hour?

  18. Some of the events will be unknown to the player directly – like “playbackStart”. The elapsed time when first frame is rendered is not guaranteed to be known. This goes to the need for a better “figure 1”, showing the actual components within the “player”, including a second figure where the “player” uses HDMI to display on a TV, or the “player” uses HDMI to push the content to a receiver which pushes frames to a television.

mlevine84 commented 5 years ago
  1. Valid comment. WG will review further.

  2. Valid comment. WG will review further.

  3. Updated PDF fixed this issue.

  4. WG will look at the definition to determine if it needs to be updated.

  5. Removed from document because it was not used in the document.

  6. Updated PDF fixed this issue.

  7. Valid point. Doc updated.

  8. Valid comment. WG will review further.

  9. Updated PDF fixed this issue.

  10. Valid point. Editing team will review this before doc is completed.

  11. WG believes repetition brings clarity to the diagram.

  12. WG agrees Media Player definition needs improvement to address the comment and provide more clarity.

  13. WG agreed and modified the definition.

  14. WG fixed typo

  15. WG has updated the scope definition for providing further clarity. However WG does not feel the set or how to define a set across eco-system is in scope for this spec.

  16. WG believes context is not a defined term. PP2 has been updated to reflect it.

  17. WG acknowledges the point raised and have added comments to clarify the definition of playbackFail event.

  18. WG agrees with the first observation "The elapsed time when first frame is rendered is not guaranteed to be known". However it is left to the analytics agent implementation to monitor the player state and other events to determine if the first frame was rendered successfully and to use videoplaybackStart event or not.

WG agrees to address the comment raised for the scenario of a streaming box (Roku) attached to a display via HDMI.

mlevine84 commented 5 years ago

It may be appropriate for Gunnar to take on comments 1 and 2. Will ask him via email if he'll accept.

mlevine84 commented 5 years ago

Prasad agreed to take comments 4 and 8.

mlevine84 commented 5 years ago

Comment 12: WG agreed to change the text to read "execution environment". Will agreed to make this change.

prasadnadig commented 5 years ago

Updated document for comment 8 (Aggregation Set); Wasn't convinced we need an update against comment 4; need to check with group.

wilaw commented 5 years ago

I completed the requested modifications to comment #12. See https://github.com/cta-standards/R4WG20-QoE-Metrics/issues/19#issuecomment-507105133 graphics for metrics doc-v2

njadia commented 5 years ago

@mlevine84 Please close this issue, all action items have been addressed.