Pathway Cloning in DASH

[mariana200196]

I wanted to revisit the pathway cloning question raised in ticket #5 because I believe that there is value in implementing it in a similar way to how it is currently done in HLS. From ticket #5, I gathered that there are three main arguments against it:

1. There is no need for pathway cloning in DASH, as in DASH a new service-location (pathway) can be introduced at any manifest update.

2. If a steering server can introduce new service locations, it creates two sources of truth describing from where content can be accessed (the manifest generator (CMS) and the steering server). This can result in divergence and unexpected problems.

3. Content sources are not ephemeral things, taking months of planning for a distributor. So there is no use case for pathway cloning, as a distributor will know about all its CDNs prior to the start of a stream.

I offer the following counterarguments:

1. While it is indeed possible to introduce a new service-location via a manifest update, this prerequires manifest manipulation. A provider would need to create a new manifest with the new pathways, then upload it to a CDN, and finally trigger a manifest update by specifying a Location@serviceLocation ID in the PATHWAY-PRIORITY list of the steering server JSON.

   This is inconvenient and a difficult process to automate when compared to pathway cloning in HLS, in my opinion.

2. While divergence cannot be ruled out, content steering's fallback option should offset any potential problems.
   If a client cannot resolve a pathway, it will simply fall back onto the default pathway. Consequently, divergence shouldn't really be a concern, I believe, and the flexibility enabled by pathway cloning would far outweigh any issues deriving from inconsistencies.

3. While not quite ephemeral in nature, I would argue that content sources are increasingly dynamic. Consider the case of local caches in ISP networks by providers such as Netflix. Would you not agree that therein lies a use case for a more adhoc and hands-off service location declaration mechanism?

   Other industry examples include the open caching project by the SVTA and emerging edge caching solutions within the area of 5G networks. One might contend that my examples raise concerns for the near future, once fledgling projects (e.g. open caching) have matured. The general argument might be true for big media companies, but not necessarily for small content-providers (e.g. startups, small businesses). The attractive prices and flexible subscriptions (quick to start and cancel) offered by CDN providers have already driven a large and ever growing number of small content-providers to host content instantly. So I do find that there are many businesses who presently have need of a convenient and on-the-fly method for advertising new service locations.

Finally, I would argue that there is value in standardising pathway cloning for HLS and DASH. If DASH decides not to support pathway cloning, a provider who wishes to introduce a new pathway mid-stream would need to program different logic to handle requests from DASH and HLS clients. This can quickly become cumbersome and annoying for developers. It would be more managable and efficient if the same logic could be used universally.

[haudiobe]

2023/02/03

@wilaw introduces the issue:

• HLS allows to add a new service location during operation.
• We say we can address this through an MPD update, but this requires that the COntent Steering providers is changing the manifest. However, this may not always be possible.
• However, for live services CDNs are pre-provisioned, adding CDNs on the fly in live is unlikely
• For VoD, this may be the case, that an MPD is produced once and content steering is used to move to a different CDN added after the MPD was produced.

@wilaw provides a feedback to @mariana200196 with a question.

[technogeek00]

I've been thinking on what bindings would be done here, dropping in the pathway cloning definition JSON for reference:

   {
       "BASE-ID": string,        // REQUIRED. Pathway ID of the Base Pathway
       "ID": string,             // REQUIRED. Pathway ID for the Pathway Clone
       "URI-REPLACEMENT":        // REQUIRED. URI replacement rules
       {
           "HOST": string,       // OPTIONAL. Hostname for cloned URIs
           "PARAMS":             // OPTIONAL. Query parameters for cloned URIs
           {
               JSON object where keys are query parameter names
               and values are query parameter values
           },
           "PER-VARIANT-URIS":   // OPTIONAL. URI overrides per Variant Stream
           {
               JSON object where keys are STABLE-VARIANT-ID strings
               and values are replacement absolute URIs
           },
           "PER-RENDITION-URIS": // OPTIONAL. URI overrides per Rendition
           {
               JSON object where keys are STABLE-RENDITION-ID strings
               and values are replacement absolute URIs
           }
       }
   }

The BASE-ID matching a target @serviceLocation is a logical connection, the application of the URI-REPLACEMENT is where I think we need to understand how it should work. The base cases would be Location and BaseURL elements that are absolute URIs, in that case the HOST and PARAMS replacements can be directly done.

The trickier application case would be a manifest that uses both attributes to achieve steering, like this:

<MPD>
    <Period>
        <Location serviceLocation="a">https://cdna.example.com/foo/bar/123456</Location>
        <Location serviceLocation="b">https://cdnb.example.com/baz/qux/123456</Location>
        <AdaptationSet>
            <Representation>
                <BaseURL>video/rep-1.mp4?param=1&param=2&param=3</BaseURL>
            </Representation>
        </AdaptationSet>
    </Period>
</MPD>

The full URI is resolved by combining the values across levels, in this case the path cloning could only target the Location attribute and event if we desired to provide a way to target specific representations for replacement, it wouldn't be possible without a @serviceLocation attached to the BaseURL object.

But maybe we just side step all of it and say if you want to use pathway cloning you are doing absolute URLs at the lowest BaseURL level?

[mariana200196]

@technogeek00 appreciate your input!

I think it's fine for pathway cloning to only target the Location attribute, or the BaseURL attribute when it is an absolute URL. In both cases, I would expect a @serviceLocation to be defined as well.

It seems reasonable to me that a new service location should follow the same path pattern as the one it wishes to clone (only with a different domain and potentially different parameters).

[wilaw]

@technogeek00 - I think in the use-case you specify, the BaseURL must carry @serviceLocation for steering to be functional. The BaseURL may be relative. The HOST is an optional parameter and would be omitted and the PARAMS substitution could still be applied.

What should we do regarding PER-VARIANT-URIS and PER-RENDITION-URIS? We could decide that these do not apply to DASH and must be omitted when steering DASH manifests.

@mariana200196 @technogeek00 here is an example manifest, steering server response and how cloning might be enacted in DASH.

<MPD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:mpeg:dash:schema:mpd:2011" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" type="dynamic" minimumUpdatePeriod="PT30S" timeShiftBufferDepth="PT30M" availabilityStartTime="2022-02-25T12:30:00" minBufferTime="PT4S" profiles="urn:mpeg:dash:profile:isoff-live:2011">
  <Location serviceLocation="1234">https://manifest-cdn1.com/</Location>
  <Location serviceLocation="5678">https://manifest-cdn2.com/</Location>
  <BaseURL serviceLocation="alpha">https://segments-cdn-A.com/</BaseURL>
  <BaseURL serviceLocation="beta">https://segments-cdn-B.com/</BaseURL>
  <Period id="Primary-Content-1">
   …
  </Period>
  <Period id="Ad-break-1">
     <BaseURL serviceLocation="ad1">https://ad-server-1.com/</BaseURL>
     <BaseURL serviceLocation="ad2">https://ad-server-2.com/</BaseURL>
     …
  </Period>
  <Period id="Primary-Content-2">
      <BaseURL serviceLocation="gamma">https://segments-cdn-C.com/</BaseURL>
      <BaseURL serviceLocation="delta">https://segments-cdn-D.com/</BaseURL>
       …
  </Period>
  <Period id="Ad-break-2">
     <BaseURL serviceLocation="ad3">https://ad-server-3.com/</BaseURL>
     <BaseURL serviceLocation="ad4">https://ad-server-4.com/</BaseURL>
      …
  </Period>
  <Period id="Primary-Content-3">
   …
  </Period>
  <ContentSteering defaultServiceLocation="1234,alpha,ad1">
      https://steeringservice.com/app?token=567
  </ContentSteering>
</MPD>

Here is the steering response

{
       "VERSION": 1,
       "TTL": 300,
       "PATHWAY-PRIORITY": ["charlie","beta","alpha","ad1","delta","5678","gamma","1234","ad4","ad3"],
       "PATHWAY-CLONES": [
           {
               "BASE-ID": "alpha",
               "ID": "charlie",
               "URI-REPLACEMENT":
               {
                   "HOST": "segments-cdn-C.com",
                   "PARAMS":
                   {
                       "token-for-charlie": "dkfs1239414"
                   }
               }
           }
       ]
   }

In this case after the first steering server response, the media player should switch to loading segments for Period "Primary-Content-1" from https://segments-cdn-C.com/...?token-for-charlie=dkfs1239414

Would this work? Any gotcha's we're not anticipating?

[mariana200196]

@wilaw thanks for the feedback!

In the example that you provided the cloned pathway "charlie" would resolve to the same URL as "gamma". I imagine this was an oversight? Assuming so, and that you meant for the steering response HOST to be different, e.g. "segments-cdn-charlie.com"

{
       "VERSION": 1,
       "TTL": 300,
       "PATHWAY-PRIORITY": ["charlie","beta","alpha","ad1","delta","5678","gamma","1234","ad4","ad3"],
       "PATHWAY-CLONES": [
           {
               "BASE-ID": "alpha",
               "ID": "charlie",
               "URI-REPLACEMENT":
               {
                   "HOST": "segments-cdn-charlie.com",
                   "PARAMS":
                   {
                       "token-for-charlie": "dkfs1239414"
                   }
               }
           }
       ]
   }

then pathway cloning in DASH would work as I envision. Meaning that the media player would switch to loading segments for Period "Primary-Content-1" from https://segments-cdn-charlie.com/?token-for-charlie=dkfs1239414.

Your example brought to my attention the issue of BaseURL duplication... Is that something that should be prohibited or should it be the responsibility of the manifest creator to ensure that BaseURLs are not duplicated?

Regarding PER-VARIANT-URIS and PER-RENDITION-URIS, I see these as more advanced use cases which could be introduced later into DASH. I think the simplest approach would be to omit them/ ignore them if they appear in DASH manifests.

[wilaw]

@mariana200196

I imagine this was an oversight? Assuming so, and that you meant for the steering response HOST to be different, e.g. "segments-cdn-charlie.com"

Exactly. Sorry about that. I edited the example very quickly just focused on the first period, and didn't notice that I had a C CDN already referenced. Your correction is what was intended.

Is that something that should be prohibited or should it be the responsibility of the manifest creator to ensure that BaseURLs are not duplicated?

Our DASH Steering spec specifically instructs the manifest to have unique @serviceLocation attributes if they intend to use those attributes for steering.

Regarding PER-VARIANT-URIS and PER-RENDITION-URIS, I see these as more advanced use cases which could be introduced later into DASH. I think the simplest approach would be to omit them/ ignore them if they appear in DASH manifests.

Agree.

[technogeek00]

The example mostly makes sense but the query parameter bit doesn't seem like it will function, the duplication would create the charlie BaseURL with the query parameters:

<BaseURL serviceLocation="charlie">https://segments-cdn-C.com.com/?token-for-charlie=dkfs1239414</BaseURL>

But deeper in a Period there would be another BaseURL or SegmentTemplate providing the actual segment description, something like:

  <Period id="Primary-Content-1">
        <AdaptationSet>
            <Representation>
                <BaseURL>video/rep-1.mp4?param=1&param=2&param=3</BaseURL>
            </Representation>
        </AdaptationSet>
  </Period>

Supposing the player is activating charlie and attempting to perform playout, URL semantics of the two levels of BaseURL would collapse the top level query parameters out:

• Top Level: https://segments-cdn-C.com.com/?token-for-charlie=dkfs1239414
• Representation Level: video/rep-1.mp4?param=1&param=2&param=3
• Resolved URL: https://segments-cdn-C.com.com/video/rep-1.mp4?param=1&param=2&param=3

So it would seem like the Representation level will require @serviceLocation which means duplication per target if you want to manipulate the parameters. I could be missing something here though.

[wilaw]

If our DASH steering logic enforces that the query arg must be applied any time the matching serviceLocation is used, taking into account any existing query args that may be present , then the player would build the URL as

https://segments-cdn-C.com.com/video/rep-1.mp4?param=1&param=2&param=3&token-for-charlie=dkfs1239414

which seems correct?

[haudiobe]

TF 2023/02/17

General support, a few updates are needed.

• It needs to be made explicit that the parameters need to be added to the URL.
• We also need to have some logic included on how to process.
• Concern on asymmetry between cache and in manifest => documentation on player behaviour needed
• HLS only allows cloning only in restricted circumstances

Please continue adding comments if needed, but we invite for adding this to the updated spec.

[haudiobe]

IOP Live TF 2023/10/3

• Implemented here: https://members.dashif.org/wg/Interoperability/document/4810?downloadRevision=5135 from @wilaw

OK for final review as part of the IOP review