Hi, I wanted to consolidate some of the discussion from this thread from jellyfin-roku as well as this thread from my own plugin repository. I would love to be able to contribute to jellyfin by working on an official trick play functionality, but obviously that requires work on every different client to accomplish, so this is mainly about trying to find a format that can work for all platform developers or at least seeing what platforms would be the easiest to compromise on. From the research I've done:
Formats:
BIF files
A standard originally made by Roku -- essentially just a stream of jpgs
Pros:
Easily created/decoded without need for third-party tools (besides already bundled ffmpeg)
Relatively easy to create client code to decode and use stream of jpgs
Used by Netflix, Disney+, Plex among others
Single network request to retrieve trick play data for an entire stream
Easily stored and moved
Cons:
Only natively supported by Roku, needs specific implementation on Swiftin (or any other native players in the future)
May require third-party tools to create? (e.g. ffmpeg + imagemagick or media-ingestion)
Cons:
Only HTML standard? May require additional implementation for native players like Roku, Swiftin
Grid of images may make that implementation less simple
HLS
Standard made by Apple for video streaming.
Pros:
Natively supported by Roku and tvOS
Cons:
Different* implementations of trick play for both Roku and tvOS
HLS.js doesn't support any trick play functionality so would also require custom impl
Clients:
tvOS/Swiftin
From what I can find on tvOS, it only natively supports using the HLS tag #EXT-X-I-FRAMES-ONLY which is an additional playlist with segments of individual frames that can be used for scrubbing. The only example I could find of this is here, where each individual preview image is its own .ts transport stream. I'll admit that all of HLS is a bit confusing to me, but if each frame would need to be its own transport stream I see that being a potentially big hassle to implement for static content and for clients that don't natively support scrubbing with #EXT-X-I-FRAMES-ONLY.
Can custom UI elements be used instead for scrubbing to support more formats?
Roku
Natively supports BIF. On their page for "trick mode" it also says SceneGraph (?) natively supports HLS. However, Roku claims on this page that Apple's approach can be less than ideal at times: "The format currently specified by HLS is a coded video I-frame. This approach is efficient because it references frame data directly from the video stream and has the added benefit of supporting digital rights management. However, additional video decoding resources are required to process these I-frames for display, which can be problematic in some cases. To make these images more accessible on a wider range of client devices, Roku supports a second playlist format, which it developed in collaboration with Disney and WarnerMedia". As such, they also support #EXT-X-TILES which are individual jpeg images that are grids containing multiple frames to be used for scrubbing. Theoretically, this means it could be interchangeably used with the images used by WebVTT.
Can custom UI elements be used instead for scrubbing to support more formats?
Web (also currently desktop + mobile)
Easiest to work with, and already has page elements for trick play images. Currently, I've already made support for BIFs but I could see WebVTT working. HLS.js (which I believe jellyfin-web uses) does not support #EXT-X-I-FRAMES-ONLY, and naturally only Roku supports #EXT-X-TILES. I'm really not sure how a custom implementation using #EXT-X-I-FRAMES-ONLY would be accomplished on the web client or on the server side as far as extracting frames goes.
Hi, I wanted to consolidate some of the discussion from this thread from jellyfin-roku as well as this thread from my own plugin repository. I would love to be able to contribute to jellyfin by working on an official trick play functionality, but obviously that requires work on every different client to accomplish, so this is mainly about trying to find a format that can work for all platform developers or at least seeing what platforms would be the easiest to compromise on. From the research I've done:
Formats:
BIF files A standard originally made by Roku -- essentially just a stream of jpgs
Pros:
Cons:
WebVTT Standard HTML web element (I think)
Pros:
Cons:
HLS Standard made by Apple for video streaming.
Pros:
Cons:
Clients:
tvOS/Swiftin From what I can find on tvOS, it only natively supports using the HLS tag #EXT-X-I-FRAMES-ONLY which is an additional playlist with segments of individual frames that can be used for scrubbing. The only example I could find of this is here, where each individual preview image is its own .ts transport stream. I'll admit that all of HLS is a bit confusing to me, but if each frame would need to be its own transport stream I see that being a potentially big hassle to implement for static content and for clients that don't natively support scrubbing with #EXT-X-I-FRAMES-ONLY.
Can custom UI elements be used instead for scrubbing to support more formats?
Roku Natively supports BIF. On their page for "trick mode" it also says SceneGraph (?) natively supports HLS. However, Roku claims on this page that Apple's approach can be less than ideal at times: "The format currently specified by HLS is a coded video I-frame. This approach is efficient because it references frame data directly from the video stream and has the added benefit of supporting digital rights management. However, additional video decoding resources are required to process these I-frames for display, which can be problematic in some cases. To make these images more accessible on a wider range of client devices, Roku supports a second playlist format, which it developed in collaboration with Disney and WarnerMedia". As such, they also support #EXT-X-TILES which are individual jpeg images that are grids containing multiple frames to be used for scrubbing. Theoretically, this means it could be interchangeably used with the images used by WebVTT.
Can custom UI elements be used instead for scrubbing to support more formats?
Web (also currently desktop + mobile) Easiest to work with, and already has page elements for trick play images. Currently, I've already made support for BIFs but I could see WebVTT working. HLS.js (which I believe jellyfin-web uses) does not support #EXT-X-I-FRAMES-ONLY, and naturally only Roku supports #EXT-X-TILES. I'm really not sure how a custom implementation using #EXT-X-I-FRAMES-ONLY would be accomplished on the web client or on the server side as far as extracting frames goes.