WebVTT files to load Tooltip Thumbnails

[adantzler]

Plyr.io looks awesome! Great work! When we encode our videos we generate a VTT file and include thumbnails in it.

We are currently using JWPlayer and it's format we follow is below:

WEBVTT

00:00.000 --> 00:05.000 /assets/preview1.jpg

00:05.000 --> 00:10.000 /assets/preview2.jpg

00:10.000 --> 00:15.000 /assets/preview3.jpg

00:15.000 --> 00:20.000 /assets/preview4.jpg

[sampotts]

Hey @adantzler,

Thanks! Nice idea. I'll have a look at implementing something for sure. :+1:

[adantzler]

That's awesome, let me know if I can help in any way.

On Tue, Mar 3, 2015 at 4:12 PM, Sam Potts notifications@github.com wrote:

Hey @adantzler https://github.com/adantzler,

Thanks! Nice idea. I'll have a look at implementing something for sure. [image: :+1:]

— Reply to this email directly or view it on GitHub https://github.com/Selz/plyr/issues/46#issuecomment-77035449.

[hookro]

I've created a little snippet to catch displayed seek onHover :dancer:

var video = document.getElementById("thumb");
video.addEventListener("loadedmetadata", getSeeker);

        function getSeeker(){

            var seekTime;

            $('.plyr__progress--seek').mousemove(function() {
                if (enableHandler) {
                    $(".plyr__tooltip.plyr__tooltip--visible").on('DOMSubtreeModified', function(){
                       seekTime = ($(this).html());
                        // add Thumbnail DIV at the requested time
                    });
                    enableHandler = false;
                }
            });

            $('.plyr__progress--seek').mouseout(function() {
                // >>remove thumbnail div correctly
            });

          timer = window.setInterval(function(){
                      enableHandler = true;               
            }, 100);
        }

[sampotts]

Cool! You might want to use MutationObserver rather than DOM events given the performance benefit: https://developer.mozilla.org/en/docs/Web/API/MutationObserver

[hookro]

Thanks! I didn't looked at performance, because I did not finished yet. I will take a look at MutationObserver, when will be ready and works well for me, I will post it here. I think it will be easy if you could provide seekTime either than grab it, but It's not a hurry.

Here is a dynamic thumbnail, I don't think its powerful, I would recommend server-side thumbnails.

HTML code

<canvas id="canvas" 
        width="750px" height="540px"
        style="display:block;">
</canvas>
<div id="screenShots"></div>

JS code

        var video = document.getElementById("thumb");
        video.addEventListener("loadedmetadata", initScreenshot)
        video.addEventListener("playing", startScreenshot);
        video.addEventListener("pause", stopScreenshot);

        var canvas      = document.getElementById("canvas");
        var ctx         = canvas.getContext("2d");
        var ssContainer = document.getElementById("screenShots");
        var videoHeight, videoWidth;
        var drawTimer   = null;

        function initScreenshot(){
            videoHeight = video.videoHeight;
            videoWidth  = video.videoWidth;

            //canvas.width  = videoWidth;
            //canvas.height = videoHeight;

        }

        function startScreenshot(){
            if(drawTimer == null) {
                drawTimer = setInterval(grabScreenshot, 1000);
            }
        }

        function stopScreenshot(){
            if(drawTimer) {
                clearInterval (drawTimer);
                drawTimer = null;
            }

        }

        function grabScreenshot(){
            ctx.drawImage(video, 0, 0, videoWidth, videoHeight);
            var img   = new Image();
            img.src   = canvas.toDataURL("image/png");
            img.width = 120;
            ssContainer.appendChild(img);
        }

Demo and explanations

[jronallo]

Just to add a couple data points on going this route to implement thumbnails over the time rail. THEOplayer and Radiant Media Player also implement thumbnails over the time rail using a WebVTT metadata file that points to images using a sprite with media fragments.

https://support.theoplayer.com/hc/en-us/articles/207460505-Preview-Thumbnails

https://www.radiantmediaplayer.com/docs/3.0/preview-thumbnails.html

When I had implemented a plugin for MediaElement.js to do similar, I found MutationObserver to help with performance.

[jamesoflol]

I started work on this today.) Just FYI in case of a freak overlap of someone else also thinking to finally do this.) Should be done some time this week https://github.com/jamesoflol/plyr/commit/5bd00abd656aa987ab69d8cd0ebac4f592d414e5

[doublex]

@jamesoflol Thanks a lot for this plugin. The issues I could find:

a) "hours" are optional in WebVTT: https://www.w3.org/TR/webvtt1/#webvtt-timestamp The problem is this regex: https://github.com/sampotts/plyr/blob/develop/src/js/plugins/previewThumbnails.js#L597

b) It should be possible to use absolute URLs (e.g. https://example.com/thumb.jpg) The problem is here: https://github.com/sampotts/plyr/blob/develop/src/js/plugins/previewThumbnails.js#L101

[doublex]

Patch to fix hours:

*** previewThumbnails.js-old    2018-12-19 19:18:36.227152525 +0100
--- previewThumbnails.js        2018-12-19 19:19:05.456949845 +0100
*************** class PreviewThumbnails {
*** 594,604 ****
              for (const line of frame.split(/\r\n|\n|\r/)) {
                  if (result.startTime == null) {
                      // The line with start and end times on it is the first line of interest
!                     const matchTimes = line.match(/(?:([0-9]{2}):)?([0-9]{2}):([0-9]{2}).([0-9]{2,3})( ?--> ?)(?:([0-9]{2}):)?([0-9]{2}):([0-9]{2}).([0-9]{2,3})/) // Note that this currently ignores caption formatting directives that are optionally on the end of this line - fine for non-captions VTT

                      if (matchTimes) {
!                         result.startTime = Number(matchTimes[1]) * 60 * 60 + Number(matchTimes[2]) * 60 + Number(matchTimes[3]) + Number("0." + matchTimes[4])
!                         result.endTime = Number(matchTimes[6]) * 60 * 60 + Number(matchTimes[7]) * 60 + Number(matchTimes[8]) + Number("0." + matchTimes[9])
                      }
                  } else {
                      // If we already have the startTime, then we're definitely up to the text line(s)
--- 594,604 ----
              for (const line of frame.split(/\r\n|\n|\r/)) {
                  if (result.startTime == null) {
                      // The line with start and end times on it is the first line of interest
!                     const matchTimes = line.match(/([0-9]{2})?:?([0-9]{2}):([0-9]{2}).([0-9]{2,3})( ?--> ?)([0-9]{2})?:?([0-9]{2}):([0-9]{2}).([0-9]{2,3})/) // Note that this currently ignores caption formatting directives that are optionally on the end of this line - fine for non-captions VTT

                      if (matchTimes) {
!                         result.startTime = Number(matchTimes[1] || 0) * 60 * 60 + Number(matchTimes[2]) * 60 + Number(matchTimes[3]) + Number("0." + matchTimes[4])
!                         result.endTime = Number(matchTimes[6] || 0) * 60 * 60 + Number(matchTimes[7]) * 60 + Number(matchTimes[8]) + Number("0." + matchTimes[9])
                      }
                  } else {
                      // If we already have the startTime, then we're definitely up to the text line(s)

[jamesoflol]

Thanks doublex! Have implemented fixes for both 'a' and 'b'. I've also fixed a minor issue with images loading out of order. https://github.com/jamesoflol/plyr/commits/preview-thumbs

@sampotts should I put in a new PR?

[doublex]

@jamesoflol Thanks a lot. Patch works for me