search

cta-wave / dpctf-tests

Repo for DPCTF Tests. We prefer to keep the Tests separated from Test Runner
Other
1 stars 5 forks source link

overlapping fragments code and test-config.json needs careful review #122

Open jpiesing opened 9 months ago

jpiesing commented 9 months ago

The code and configuration data for overlapping_fragments needs a careful review.

1) test-config.json appears to have duplicate identical playout[] arrays for each framerate family (i.e. 6 instead of 3). 2) The spec in 8.10.4 says;

For a track buffer that supports a media profile, Overlapped Fragment Playback of a CMAF Switching Set, consisting of K tracks refers to the following actions, starting with k=1 for the track with lowest quality to k=K with the highest quality:

The playout array uses ss1 and steps from [1,1,1] to [1.1.4] then [1,2,5] to [1,2,8] and so on. However the Representations in ss1 are not in quality order. Looking at the MPD, I might expect the following to give increasing quality order ...

3) When I look at a recording of this running, I see the following representations;

FritzHeiden commented 8 months ago

Thanks for pointing that out, I will fix the playout parameters accordingly.

jpiesing commented 8 months ago

Thanks for pointing that out, I will fix the playout parameters accordingly.

The results from running this test in July were worse than many of the other tests. I really recommend not just running it but very carefully checking that it plays the media according to playout[] in the correct order and for the correct time.

gitwjr commented 7 months ago

@FritzHeiden noted a change to the playout parameters were needed. He will add additional comments on what he has done and its status.

FritzHeiden commented 7 months ago

As @jpiesing already suggested and what can be seen in the SS1 MPD, representations are not ordered by quality, so the playout parameter needs to be adjusted accordingly. Doing so results in the correct behaviour of playback of representations in order of increasing quality.

As the specification suggest, there should be multiple tracks which should be played in increasing order, however, the sparse matrix for video content specifies the overlapping fragments test to use t1 content, which only has one track. Should I change the test to use SS1 content?

gitwjr commented 6 months ago

Fraunhofer will test again on 2 TVs in the lab. If the results are OK, close this issue.

jpiesing commented 5 months ago

Fraunhofer will test again on 2 TVs in the lab. If the results are OK, close this issue.

@louaybassbouss @FritzHeiden Did you do these tests as mentioned before Christmas?

louaybassbouss commented 4 months ago

As @jpiesing already suggested and what can be seen in the SS1 MPD, representations are not ordered by quality, so the playout parameter needs to be adjusted accordingly. Doing so results in the correct behaviour of playback of representations in order of increasing quality.

As the specification suggest, there should be multiple tracks which should be played in increasing order, however, the sparse matrix for video content specifies the overlapping fragments test to use t1 content, which only has one track. Should I change the test to use SS1 content?

@jpiesing as @FritzHeiden mentioned, t1 content (see sparse matrix) is not suitable for overlapping fragments since it contains only a single representation. Even the test passes, this does not mean it is logically correct since the current playout using only a single representation (because t1 has only one presentation). We need content with multiple presentations and @FritzHeiden proposed ss1. Do you agree?

jpiesing commented 4 months ago

As @jpiesing already suggested and what can be seen in the SS1 MPD, representations are not ordered by quality, so the playout parameter needs to be adjusted accordingly. Doing so results in the correct behaviour of playback of representations in order of increasing quality. As the specification suggest, there should be multiple tracks which should be played in increasing order, however, the sparse matrix for video content specifies the overlapping fragments test to use t1 content, which only has one track. Should I change the test to use SS1 content?

@jpiesing as @FritzHeiden mentioned, t1 content (see sparse matrix) is not suitable for overlapping fragments since it contains only a single representation. Even the test passes, this does not mean it is logically correct since the current playout using only a single representation (because t1 has only one presentation). We need content with multiple presentations and @FritzHeiden proposed ss1. Do you agree?

Yes I agree. @louaybassbouss @FritzHeiden do you want me to make a proposal in the sparse matrix that you will use or would you prefer to make something useful and me to update the sparse matrix to match?

jpiesing commented 4 months ago

February 20th meeting: Fraunhofer to go ahead and use ss1. Documentation can then be adjusted afterwards.

jpiesing commented 4 months ago

March 5th Everything behaves correctly and looks good with SS1 but didn't do any in depth investigation. This was run in the plugfest last week but recordings are yet to be analysed. Leave it open until those recordings have been run with the OF and close it if things still look good.

jpiesing commented 3 months ago

In the London plugfest;

jpiesing commented 2 months ago

In the London plugfest;

* the "[OF] Video: The presented sample shall match the one reported by the currentTime value within the tolerance of +/-(2/framerate + 20ms)" observation failed on all 8 TV sets.

This is addressed by the solution to https://github.com/cta-wave/device-playback-task-force/issues/123 .

* the "[OF] Video: The playback duration shall match the duration of the CMAF Track" observation failed on 6 of 8 TVs

On the 3 better TVs from the London Plugfest, the duration failures are due to missing frames, which are below the failure threshold in 2 of 3 cases.

I recommend we close this when https://github.com/cta-wave/device-playback-task-force/issues/123 is closed.