search

cta-wave / dpctf-tests

Repo for DPCTF Tests. We prefer to keep the Tests separated from Test Runner
Other
1 stars 5 forks source link

Extra test information required by Observation Framework for splice tests #32

Closed andyburras closed 3 years ago

andyburras commented 3 years ago

The Device Observation Framework will require some additional information provided by the Test Runner JSON for the splice test observations. For flexibility, we propose -

For each sequential CMAF Presentation in a WAVE Program:

louaybassbouss commented 3 years ago

@andyburras fragment_duration is already extracted (see /generated/tests.json). What do you mean by first_frame_number and last_frame_number?

andyburras commented 3 years ago

@louaybassbouss Currently there is only a single fragment_duration given e.g.:

        "61f355e4382099592033eb547dfc0608": {
            "path": "avc/playback-over-wave-baseline-splice-constraints-manual__stream__.html",
            "video": "https://dash.akamaized.net/WAVE/vectors/avc_sets/1/stream.mpd",
            "audio": "",
            "code": "playback-over-wave-baseline-splice-constraints-manual.html",
            "config": {
                "cmaf_track_duration": "PT1M0.0S",
                "fragment_duration": "PT2.0S"
            }
        },

but (I think) the spliced content could have different fragment durations from the original content so we need it for each sequential CMAF Presentation.

We need to know what are the expected frames to be presented for each portion of spliced content, e.g. a post-roll advert scenario could have content X -> content Y -> content Z. A mid-roll advert scenario could have content X -> content Y -> content X. At the splice points we need to confirm that the correct last frame of each portion was displayed followed by the correct initial frame of the following portion.

louaybassbouss commented 3 years ago

@louaybassbouss Currently there is only a single fragment_duration given e.g.:

        "61f355e4382099592033eb547dfc0608": {
            "path": "avc/playback-over-wave-baseline-splice-constraints-manual__stream__.html",
            "video": "https://dash.akamaized.net/WAVE/vectors/avc_sets/1/stream.mpd",
            "audio": "",
            "code": "playback-over-wave-baseline-splice-constraints-manual.html",
            "config": {
                "cmaf_track_duration": "PT1M0.0S",
                "fragment_duration": "PT2.0S"
            }
        },

but (I think) the spliced content could have different fragment durations from the original content so we need it for each sequential CMAF Presentation.

@andyburras in the example you provided there is one fragment_duration for all representations, do you expect different fragment_durations for each presentation (or within the same presentation)? In issue #22 there is a discussion about this.

We need to know what are the expected frames to be presented for each portion of spliced content, e.g. a post-roll advert scenario could have content X -> content Y -> content Z. A mid-roll advert scenario could have content X -> content Y -> content X. At the splice points we need to confirm that the correct last frame of each portion was displayed followed by the correct initial frame of the following portion.

Agree but who can provide this information? within the test implementation we don't have access to these parameters. You can provide any additional configuration you need in the OF by overriding the test-config.json as described in the dpctf-deploy#mapping-new-content-into-the-container. @yanj-github you may already used this option.

andyburras commented 3 years ago

do you expect different fragment_durations for each presentation (or within the same presentation)?

As far as I can tell from the WAVE Baseline Splice Profile Constraints, the fragment time needs to be aligned at the Splice Points. But provided that requirement is met I can't see it saying that the fragment duration of the spliced content needs to be the same as the original content? If they can be different then there will need to be an entry in the JSON for each CMAF Presentation in the WAVE Program. So we need to raise that query, maybe with CSTF.

Agree but who can provide this information?

The author of any given test will need to add the expected result data to that test's configuration.

louaybassbouss commented 3 years ago

@andyburras regarding fragment_duration we can provide it on presentation level but we need first to have the content. As discussed in #22, it will be multi-period content. Lets discuss this during our meeting later today.

jpiesing commented 3 years ago

April 13th meeting Propose to add an array to the test configuration file where each item corresponds to a period in the MPD & contains the fragment duration of the period & the overall duration of the period.

FritzHeiden commented 3 years ago

I extended the generate script to extract duration of each representation and include it into tests.json. The generated directory was also updated to reflect this.

andyburras commented 3 years ago

@FritzHeiden Bit confused as to how this is intended to be used within a test. The durations are for the video Representations within a Period, i.e.:

            "video": "http://dash.akamaized.net/WAVE/ContentModel/SinglePeriod/Fragmented/ToS_MultiRate_fragmented.mpd",
            "audio": "https://dash.akamaized.net/WAVE/ContentModel/SinglePeriod/Fragmented/ToS_HEAACv2_fragmented.mpd",
            "code": "playback-over-wave-baseline-splice-constraints.html",
            "config": {
                "mediaPresentationDuration": "PT0H12M12.167S",
                "maxSegmentDuration": "PT0H0M2.167S",
                "representations": {
                    "video1/1": {
                        "duration": 732
                    },
                    "video1/2": {
                        "duration": 732
                    }

for an MPD with:

<Period duration="PT0H12M12.167S">
[...]
<Representation id="video1/1" width="1920" height="800" frameRate="24" sar="1:1" bandwidth="1016073"/>
<Representation id="video1/2" width="1920" height="800" frameRate="24" sar="1:1" bandwidth="2029757"/>
</AdaptationSet>
</Period>

whereas I was expecting there to either be multiple Periods, or multiple MPDs? Will the test be playing the Representations one after another to mimic a splice use case, i.e. we can assume that any listed "config"."representations" will be played in the order listed, for their full "duration", with each Representation begining at mezzanine frame 1?

louaybassbouss commented 3 years ago

@andyburras as discussed in https://github.com/cta-wave/dpctf-tests/issues/22#issuecomment-795091964 this will be multi-period content (3 periods each with a single representation). If you see it is better to provide the duration on period level and not on representation, we can do that as well. wdyt?

andyburras commented 3 years ago

@louaybassbouss The O.F. needs to be able to calculate the expected last frame number and first frame number of each piece of displayed video. I'm not clear how the test running on the device will operate, will it simply play a multiple period MPD end-to-end, so for e.g. an ad insertion test would have main-content-period => advert-period => main-content-period? In which case we would need the duration of each period that the test is to display. Can we can assume that the main-content and the advert will both start at frame number 1? And can we assume that if there are three periods then the third will begin at first period's last frame number +1, i.e. three period tests will always be an advert insertion use case? (if not then we would require some further test configuration info).

louaybassbouss commented 3 years ago

@louaybassbouss The O.F. needs to be able to calculate the expected last frame number and first frame number of each piece of displayed video. I'm not clear how the test running on the device will operate, will it simply play a multiple period MPD end-to-end, so for e.g. an ad insertion test would have main-content-period => advert-period => main-content-period?

@andyburras yes this is the plan. The Test runner will simply plays the 3 periods end-to-end .

In which case we would need the duration of each period that the test is to display. Can we can assume that the main-content and the advert will both start at frame number 1?

yes (@rbouqueau @dsilhavy correct me if I am wrong)

And can we assume that if there are three periods then the third will begin at first period's last frame number +1, i.e. three period tests will always be an advert insertion use case? (if not then we would require some further test configuration info).

yes (@rbouqueau @dsilhavy correct me again if I am wrong).

yanj-github commented 3 years ago

@louaybassbouss I dont see "tests.json" file specified 3 periods duration for a splicing test as a part of shared parameters. OF required these 3 periods duration to calculate the splicing point. Could you let me know where these parameters are determined please?

yanj-github commented 3 years ago

@louaybassbouss We need to know the strcuture of the 3 periods duration in "tests.json" file to implement the parameter parsing. However, current from https://github.com/cta-wave/dpctf-tests/blob/master/generated/tests.json there isnt any splicing test is defined. Can you kindly let us know what are these 3 periods duration looks like from the "tests.json" file, if we would have splicing test defined please?

louaybassbouss commented 3 years ago

@louaybassbouss We need to know the strcuture of the 3 periods duration in "tests.json" file to implement the parameter parsing. However, current from https://github.com/cta-wave/dpctf-tests/blob/master/generated/tests.json there isnt any splicing test is defined. Can you kindly let us know what are these 3 periods duration looks like from the "tests.json" file, if we would have splicing test defined please?

Lets say we have 3 Periods and each with a single video representation (which means in total we have 3 representations e.g. with IDs "1", "2" and "3"). The config of the corresponding test will look like this:

"config": {
        ....
        "representations": {
          "1": {
            "duration": 6.72,
            "fragment_duration": 0.96
          },
          "2": {
            "duration": 1.92,
            "fragment_duration": 0.96
          },
         "3": {
            "duration": 6.72,
            "fragment_duration": 0.96
          }
        }
      }
andyburras commented 3 years ago

Is that not a single Period with 3 Representations? What would it look like if say there were 2 Periods, the first with 3 Representations and the second with 2?

louaybassbouss commented 3 years ago

Is that not a single Period with 3 Representations? What would it look like if say there were 2 Periods, the first with 3 Representations and the second with 2?

@andyburras it is the same structure. you always have a list of representation in the json config. If we have more that one representation in a period (as in your example), we need then additional configuration which tells the test which representation from each period to play. But we don't have such tests. What we can do is to extend the structure and add a new period property like this if it helps (period numbering start with 1): 

"config": {
        ....
        "representations": {
          "1": {
            "duration": 6.72,
            "fragment_duration": 0.96,
            "period": 1
          },
          "2": {
            "duration": 1.92,
            "fragment_duration": 0.96,
            "period": 2
          },
         "3": {
            "duration": 6.72,
            "fragment_duration": 0.96,
            "period": 3
          }
        }
      }
andyburras commented 3 years ago

@louaybassbouss Ah, I think I may be seeing the piece I was missing. So in your previous example the tests.json would be e.g.

"config": {
        ....
        "representations": {
          "1": {
            "duration": 6.72,
            "fragment_duration": 0.96
          },
          "2": {
            "duration": 1.92,
            "fragment_duration": 0.96
          },
         "3": {
            "duration": 6.72,
            "fragment_duration": 0.96
          }
        }
      }

and the test_config.json for a splicing test would have e.g.:

  "<splicing-test-name>": {
    "playout": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, ...]
  },

So that would allow the Observation Framework to work out the splicing point?

yanj-github commented 3 years ago

@louaybassbouss and @FritzHeiden can you kindly confrim if the splicing test will define playout: [list of representation number] from test-config.json to allow OF to calculate the duration based on fragment_duration tests.json?

FritzHeiden commented 3 years ago

@yanj-github You should be able to do this, however, I did not test this yet. Furthermore, the playout parameter is used to specify which representation to use, if there are multiple in the same period. If there are multiple periods with only one representation each, the representations will be played consecutively and the playout parameter is not needed.

yanj-github commented 3 years ago

Thanks @FritzHeiden I have some more question if you could help please?

louaybassbouss commented 3 years ago

@yanj-github @jpiesing I would like to discuss this during our call today. This is the definition of playout parameter for "8.8 Playback over WAVE Baseline Splice Constraints".

playout[i]: Provides the triple (Switching Set, CMAF track number, Fragment number) for every playout position i=1,…,N that is be played out

We decided to use Multi-Period Content for this test (see https://github.com/cta-wave/dpctf-tests/issues/22#issuecomment-795091964) we don't see a need for a playout parameter for this test.

louaybassbouss commented 3 years ago

@yanj-github can we close this issue?

yanj-github commented 3 years ago

This is tracked on #22 we can close this as this is duplicated.