QoE Metrics Report: Behavior after update of the Service Access Information

[dsilhavy]

Feature description

In #30 and #31 we discussed how to periodically check for updates of the Service Access Information. This issue discusses changes of ServiceAccessInformation.clientMetricsReportingConfigurations and their impact on running streaming sessions.

Scenarios

In the following different scenarios are depicted in which clientMetricsReportingConfigurations is modified. This includes addition of new configurations and removal of existing configurations. For updates of existing configurations, all attributes of ServiceAccessInformation.clientMetricsReportingConfigurations are listed with the expected behavior of the Media Session Handler after such an attribute has changed.

Addition of a new metrics reporting configuration

A new configuration is added. It is handled similar to the configurations at playback start. A metrics reporting timer is started, and the metrics are requested periodically.

Removal of an existing metrics reporting configuration

An existing configuration is removed. Stop the reporting for the specific metricsReportingConfigurationId.

Update of an existing metrics reporting configuration

One or more of the following attributes are changed.

Attribute	Expected Behavior
metricsReportingConfigurationId	Counts as a new configuration, see above.
serverAddresses	tbd
scheme	If this is a valid change, we need to check if the scheme is supported by the Media Stream Handler. If it is supported, we start a timer to periodically request the metrics defined in metrics. In addition, we stop the timer for the old scheme
dataNetworkName	Change the data network name when sending the metrics reports. Not yet implemented
reportingInterval	Adjust the reporting interval based on the new value. Should probably take effect after the previous metrics timer iteration has finished. Alternatively, we can stop the current timer and restart with a new value.
samplePercentage	If the value is increased we could re-evaluate. For instance, if we have ten clients, five of them are reporting and sample percentage is increased from 50.0 to 80.0: We can re-evaluate the five clients that are not reporting. In this case 60% of the remaining five clients should now report as well. If the value is set to zero, we stop reporting. If the value is decreased and higher than zero: Should we re-calculate and stop reporting for some of the clients?
urlFilters	Adjust the filter that is applied. Not yet implemented
metrics	Adjust the request to the Media Stream Handler for the target metrics

[dsilhavy]

Basic support for reacting to Service Access Information changes has been added:

• If a clientMetricsReportingConfiguration is removed from the ServiceAccessInformation the reporting timer is terminated
• If the list of serverAddresses is empty stop the reporting timer
• If the samplePercentage is set to 0 stop the reporting timer
• If the samplePercentage is set to 100 start the reporting timer

[rjb1000]

Just to probe the behaviour of your 5GMS Client implementation a bit: if the Media session Handler notices that samplePercentage has changed from 0% to 100%, would it already have an up-to-date set of QoE metrics ready to report immediately, or would it need to wait to receive some from the Media Stream Handler on the next sampling cycle?

Or, to ask the same question another way: does the Media Stream Handler sample QoE metrics according to the configured sampling frequency and notify them to the Media Session Handler regardless of whether reporting is currently enabled or disabled by the samplePercentage configuration property?

In other words, it seems valid for the Media Session Handler to receive and store QoE metrics (perhaps up to some buffer limit) but only forward QoE metrics reports to the 5GMS AF when (independently) configured to do so.

So, in implementation terms that's two different asynchronous activities:

• The Media Stream Handler samples and notifies metrics at the configured sampling frequency.
• The Media Session Handler reports metrics at the configured reporting frequency.

[dsilhavy]

Thanks @rjb1000 I think this raises some interesting discussion points.

Current behavior

Right now, the behavior of persisting the QoE Metrics Reporting data is located in the Media Stream Handler. This is the default workflow:

Workflow - Default

1. A new playback session is initialized.
2. The Media Stream Handler instantiates a new QoeMetricsReporter (e.g. the QoeMetricsReporterExoplayer : https://github.com/5G-MAG/rt-5gms-media-stream-handler/pull/67/files#diff-d1556c694c8ad1ae1c05710bd0c9aab2c0785081766662d7fe14a212ae9fcbec) for each supported metricsReportingConfigurationId (in case the corresponding scheme is supported)
3. Each instance of a QoeMetricsReporter starts collecting metrics. Note that currently most of our metrics are event driven, for instance a RepresentationSwitch is linked to a specific event in the Exoplayer and is not derived based on the samplingPeriod
4. Whenever the collected QoE metrics are dispatched to the Media Session Handler the list of persisted QoE metrics in the Media Stream Handler is cleared. This dispatching happens typically upon concrete requests by the Media Session Handler mainly based on the reportingInterval timer. In addition, the QoE metrics are also dispatched to the Media Session Handler when a streaming session has ended (e.g. playback terminates or a new source stream is selected).
5. The Media Session Handler decides whether to dispatch this information to the selected AF server address or not. If samplePercentage is 0 we can simply ignore all incoming QoE metrics data from the Media Stream Handler.

Workflow - Change of sample percentage

Now in the example you mentioned above in which samplePercentage is 0% the QoE metrics are still derived in the Media Stream Handler. Since they are never dispatched to the Media Session Handler unless the streaming session ends, they queue up. As you mentioned, there should probably be a buffer limit at which we decide to discard parts of the data. This also links to the discussion item below: How much data do we persist in case reportingInterval is not defined? What shall be included in the "single report" in the end?

Coming back to your example, if the samplePercentage is changed from 0% to 100% the Media Session Handler now immediately requests QoE metrics information from the Media Stream Handler. The Media Stream Handler dispatches all the collected information (basically everything since playback start) resulting in a large XML payload. Only then the list of collected QoE metrics is cleared in the Media Stream Handler.

Discussion Items

Persisting the data

How long should we persist the QoE metrics in the Media Stream Handler? For instance, if reporting interval is not defined 26.512 says:

When this property is omitted, a single final report shall be sent immediately after the media streaming session has ended.

Does this mean we collect everything from the start until the playback session has ended? Also, what happens if we play a livestream for multiple hours? This will certainly crash the client at some point as the memory consumption keeps on increasing.

I might have a misunderstanding of the newly added samplingPeriod. So far, I thought this is a value that tells the Media Stream Handler how often to query non-event based metrics. For instance, if samplingPeriod is set to 5 seconds it tells the Media Stream Handler to query the buffer level every 5 seconds and persists this data. Reading your last comment, I am thinking that the samplingPeriod could also be used to tell the Media Stream Handler to only persists data that is X seconds old. For instance, if samplingPeriod is set to 5 seconds I am removing metrics that are older than 5 seconds before dispatching them to the Media Session Handler.