Switching set test and overlapping test

[yanj-github]

I have tried to run a switching set test and a overlapping test. And noticed these doesnt match what I was expected. Maybe I am wrong, correct me please if I was wrong please.

At the moment, the switching set test switches from representation 0 to 7 in order, so it switches 7 times. Where I was thinking it should switch once according to the configuration. https://github.com/cta-wave/dpctf-tests/blob/master/test-config.json I also noticed the configuration is not defined for the avc group but I can find one for Folder1/switching-set-playback-manualToS_MultiRate_fragmentedToS_HEAACv2_fragmented.html Is this might be the reason? I was running avc/switching-set-playback-manual....

The overlapping test does the same that it switches from representation 0 to 7. However, I was thinking this should be appear to be same as sequential-playback, so that only one representation should be played back.

[FritzHeiden]

The test-config.json that is present in the repository only has the playout parameter configured for Folder1/switching-set-playback-manual__ToS_MultiRate_fragmented__ToS_HEAACv2_fragmented.htmland Folder2/playback-over-wave-baseline-splice-constraints-manual__ToS_MultiRate_fragmented__ToS_HEAACv2_fragmented.html, as you already noticed.
For every other test, where there is no playout parameter defined, the playout parameter is automatically determined by dividing the video duration by the number of representations. So the behaviour you observe is correct. If you only want representations 0 and 1 to be used in the test, you have to specify the playout parameter accordingly in the test-config.json.
The dpctf-deploy projects docker-compose.ymlfile (https://github.com/cta-wave/dpctf-deploy/blob/master/docker-compose.yml) already has the configuration necessary for mapping custom test-config.json files in it at the appropriate place as a comment. Place a custom test-config.json within the dpctf-deploy projects directory and uncomment the according line to use it in the container.

[yanj-github]

Thanks @FritzHeiden that is clear to me, I will try to define playout for avc see if that will work. Can you answer my 2nd question about the overlapping test please? It this test should switch that taking playout same as switching set test or it should be same as 8.2 sequential-playback? What I can see now is that this test switches from representation 0 to 7.

[yanj-github]

@FritzHeiden I have changed configuration file added the following: "avc/switching-set-playback-manualstream.html": { "playout": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], "total_representations": 2, "duration": 60 }, But nothing is playedback for swtiching set test and the test shows ERROR. Did I set anything wrong or do I need to change anything else please?

[FritzHeiden]

According to the specification, the overlapping test is supposed to switch between representations.

Regarding your test error, the test_representationsand durationparameters are compared against the values extracted from the mpd. If they mismatch, an error is thrown. I updated the tests and test runner to properly communicate occurring errors, by updating the qr code accordingly, as well as including the error message into the test results.

[yanj-github]

Thanks @FritzHeiden.

According to the specification, the overlapping test is supposed to switch between representations.

OK this is fine, but OF required to know the playout same as we have for the switching set. Will test runner read playout parameters from the configuration file if it is defined please?

Regarding your test error, the test_representationsand durationparameters are compared against the values extracted from the mpd. If they mismatch, an error is thrown. I updated the tests and test runner to properly communicate occurring errors, by updating the qr code accordingly, as well as including the error message into the test results.

Thanks for the information, I still have some questions as follows: total_representations is number of totable representations in mpd? duration should match the one defined in mpd, is this in seconds please? Also I think this is duplicated information as we have duration stated in tests.json file as well. playout list should contain representation id and the list number should match entire duration which is durarion / fragment_duration?

[yanj-github]

I didn't defined "duration" which works fine. Otherwise, "duration": 60 keep failing with error duration mismatch. OF reads duration from tests.json so for OF "duration" in test-config.json does not matter.

[FritzHeiden]

Will test runner read playout parameters from the configuration file if it is defined please?

A test will read the playout parameter if the specification defines it.

total_representations is number of totable representations in mpd?

Yes.

duration should match the one defined in mpd, is this in seconds please?

The duration is in seconds, yes. I updated the documentation accordingly.

playout list should contain representation id and the list number should match entire duration which is durarion / fragment_duration?

We implemented the data structure for the playout parameter according to the specification

I didn't defined "duration" which works fine. Otherwise, "duration": 60 keep failing with error duration mismatch. OF reads duration from tests.json so for OF "duration" in test-config.json does not matter.

I updated the error message to contain the expected value so it is easier to see what the exact mismatch is. If you don't require the duration value from the test-config.json, you can leave it out as it is an optional parameter for executing the tests.

[yanj-github]

Thanks @FritzHeiden I cannot see the expected duration value but I guess I might need the lates change of code. This is fine for me I will take the duration out for now which works fine to me.

Do you have documnetation about test-config.json please? Apologies if you already posted but I cannot find it now. Can you let me know where I can find it please?

[FritzHeiden]

Can you let me know where I can find it please?

It sits next to the README.md: TEST_CONFIG.md

[yanj-github]

Thanks @FritzHeiden I am happy to close this issue.