[Feature request] Thumbnails in DASH

[iBicha]

Is your feature request related to a problem? Please describe. I'm currently building a Roku frontend for Invidious (Playlet) Roku expects thumbnails, aka storyboards (for "trick play" or scrubbing through the progress bar) to be included in the DASH-IF manifest (DASH-standard thumbnail tiles) So it would be cool to have an option where the storyboard included in the dash manifest, so that Roku can consume it properly

Describe the solution you'd like /api/manifest/dash/id/:videoid would return a DASH manifest that has the storyboard info included (like this example)

Describe alternatives you've considered I've tried to feed the storyboard manually into the system, and I did not succeed. My options are to recreate the thumbnail data somehow (which is not trivial) or generate an extra <AdaptationSet> section for the thumbnails in the DASH manifest. The latter is the most doable solution, but it's still a bad solution because

• I will have to serve the DASH manifest manually in a local server, proxying the request for the manifest, and injecting the thumbnails section
• I should not modify the DASH manifest on the fly like that, because I risk breaking it

Additional context Is this feature is to be considered (I hope) maybe it could be an option like /api/manifest/dash/id/:videoid?storyboard=true so that the feature remains backward compatible with existing API

[iBicha]

Well, this is probably going to be low priority to folks. I'm trying to figure out how to transform the storyboards into a correct section in the MPD. But I'm not that familiar with DASH. A reference on the data transformation would be great!

[SamantazFox]

I'm not really familiar with DASH either, especially given that it's a closed standard.

Maybe try looking on the interwebs for existing DASH manifests with storyboards? That's generally what I do ^^

[iBicha]

I'm not really familiar with DASH either, especially given that it's a closed standard.

Maybe try looking on the interwebs for existing DASH manifests with storyboards? That's generally what I do ^^

After looking around, I figured that the standard gets developed mostly here https://github.com/Dash-Industry-Forum/DASH-IF-IOP with a reference implementation here https://github.com/Dash-Industry-Forum/dash.js

I realized that this might be super low priority for most people since it's a niche needed feature, I took a stab at it. I'll just dump everything in here in case someone comes digging for this stuff someday.

After lots of trial and error, I kinda made it work, by transforming this storyboard object

"storyboards":[
      {
         "url":"/api/v1/storyboards/D30r0CwtIKc?width=48&height=27",
         "templateUrl":"https://i.ytimg.com/sb/D30r0CwtIKc/storyboard3_L0/M$M.jpg?sqp=-oaymwENSDfyq4qpAwVwAcABBqLzl_8DBgif-JWbBg%3D%3D&sigh=rs%24AOn4CLDlgMqul6laAfofpsEja844drO4pw",
         "width":48,
         "height":27,
         "count":100,
         "interval":0,
         "storyboardWidth":10,
         "storyboardHeight":10,
         "storyboardCount":1
      },
      {
         "url":"/api/v1/storyboards/D30r0CwtIKc?width=80&height=45",
         "templateUrl":"https://i.ytimg.com/sb/D30r0CwtIKc/storyboard3_L1/M$M.jpg?sqp=-oaymwENSDfyq4qpAwVwAcABBqLzl_8DBgif-JWbBg%3D%3D&sigh=rs%24AOn4CLBr_-tPfrPRAcfA7HIjlIrY3Gw-Ig",
         "width":80,
         "height":45,
         "count":65,
         "interval":1000,
         "storyboardWidth":10,
         "storyboardHeight":10,
         "storyboardCount":1
      },
      {
         "url":"/api/v1/storyboards/D30r0CwtIKc?width=160&height=90",
         "templateUrl":"https://i.ytimg.com/sb/D30r0CwtIKc/storyboard3_L2/M$M.jpg?sqp=-oaymwENSDfyq4qpAwVwAcABBqLzl_8DBgif-JWbBg%3D%3D&sigh=rs%24AOn4CLACRYSDqkCU-R0D70XwxADh2UCjcg",
         "width":160,
         "height":90,
         "count":65,
         "interval":1000,
         "storyboardWidth":5,
         "storyboardHeight":5,
         "storyboardCount":3
      }
   ],

Into this (the first storyboard object in the json is discarded as the interval is 0)

<AdaptationSet id="4" mimeType="image/jpeg" contentType="image">
    <SegmentTemplate media="https://i.ytimg.com/sb/D30r0CwtIKc/storyboard3_L1/M$Number$.jpg?sqp=-oaymwENSDfyq4qpAwVwAcABBqLzl_8DBgif-JWbBg%3D%3D&sigh=rs%24AOn4CLBr_-tPfrPRAcfA7HIjlIrY3Gw-Ig" duration="70" startNumber="0" />
    <Representation id="thumbnails_4" bandwidth="1800" width="800" height="315">
        <EssentialProperty schemeIdUri="http://dashif.org/guidelines/thumbnail_tile" value="10x7" />
    </Representation>
</AdaptationSet>
<AdaptationSet id="5" mimeType="image/jpeg" contentType="image">
    <SegmentTemplate media="https://i.ytimg.com/sb/D30r0CwtIKc/storyboard3_L2/M$Number$.jpg?sqp=-oaymwENSDfyq4qpAwVwAcABBqLzl_8DBgif-JWbBg%3D%3D&sigh=rs%24AOn4CLACRYSDqkCU-R0D70XwxADh2UCjcg" duration="25" startNumber="0" />
    <Representation id="thumbnails_5" bandwidth="7200" width="800" height="450">
        <EssentialProperty schemeIdUri="http://dashif.org/guidelines/thumbnail_tile" value="5x5" />
    </Representation>
</AdaptationSet>

Which I shove into the <Period> XML node of the DASH, before serving it to the TV.

This made the thumbnails show up in the default viewer, which is a win

The problem is that this setup does not account for the last page which might have a different size than the rest of the pages. Because of that, all the thumbnails look fine except the last page towards the end.

But I can live with it. Implementation currently in https://github.com/iBicha/playlet/pull/15 (more specifically) (warning, written badly in brightscript)

[iBicha]

Closing due to https://github.com/iv-org/invidious/pull/3987#issuecomment-1763126503 Please reopen if there's demand for storyboards in DASH on other platforms