Control playback of media from speakerConsole

[complanar]

When using video or audio in an impress.js presentation it is very difficult to control the playback of them, because media does not respond to clicks at the console window. At the moment I have to focus the presentation window and start/pause/seek/toggle fullscreen there, which is very difficult if the presentation view is located behind you.

That problem applies to all interactive elements in the presentation I think, for example filling forms, hovering elements and so on.

I guess the is a great effort necessary to propagate all those events to the presentation window and trigger the corresponding actions. But that would increase the usability very much.

[complanar]

Just realized, that this applies to the toolbar, too. The speaker console reacts to navigation commands in the presentation view but not vice versa. That seems a little weird. It should be just the other way round.

[henrikingo]

Well, the speaker command of course has its own navigation buttons. But for everything else, you have a point. It could be a good idea to render the toolbar also inside the speaker console. Would be interesting to try.

For media, I think there's definitively a need for a plugin to support media objects: start automatically when entering a step, stop when leaving, integrate with substep, and so on. (And even then, the issue of manually starting and stopping from the console is still valid.)

[complanar]

Yes, I know the navigation buttons of the speaker console, of course. But if the toolbar would work the other way round, we would not need them, do we?

Starting and pausing media when entering or leaving a step should be easy with hooking into the stepenter and stepleave events. It would be also useful to add a class to the body tag when playing a video, for maybe darkening the background in the meanwhile with css. This class should be removed, when the media ended or the step is left. This is my current quick and dirty script:

    (function (document, window) {
      "use strict";
      //var lib;

      var mediatools = window.mediatools = function () {

        var steps = document.getElementsByClassName('step');
        for (var i = 0; i < steps.length; i++) {
          if (steps[i].classList.contains('videoslide')) {
            steps[i].addEventListener('impress:stepenter', function() {
              document.body.classList.add('onvideoslide');
            });
            steps[i].addEventListener('impress:stepleave', function() {
              document.body.classList.remove('onvideoslide');
            });
          }
          /* Pause audio and video automatically when leaving their step. */
          steps[i].addEventListener('impress:stepleave', function() {
            var videos = this.getElementsByTagName('video');
            for (var k = 0; k < videos.length; k++) {
              videos[k].pause();
              document.body.classList.remove('videoplaying');
            }
            var audios = this.getElementsByTagName('audio');
            if (audios != undefined) {
              for (var k = 0; k < audios.length; k++) {
                audios[k].pause();
              }
            }
          });
        }

        /* parse html5media divs: <div class="html5media" data-source="PATH" data-type="MIMETYPE"></div> */
        var parseMedia = function (parentElement) {
          var mime = parentElement.getAttribute('data-type');
          var mimeParts = mime.split('/');
          var path = parentElement.getAttribute('data-source');
          parentElement.innerHTML = '<' + mimeParts[0] + ' controls><source src="' + path + '" type="' + mime +'"> Your Browser does not support HTML5 ' + mimeParts[0] + '.</' + mimeParts[0] + '>';
        }

        var media = document.getElementsByClassName('html5media');
        for (var i = 0; i < media.length; i++) {
          parseMedia(media[i]);
        }

        var videos = document.getElementsByTagName('video');
        for (var i = 0; i < videos.length; i++) {
          videos[i].addEventListener("playing", function() {
          document.body.classList.add('videoplaying');
          });
          videos[i].addEventListener("play", function() {
          document.body.classList.add('videoplaying');
          });

          videos[i].addEventListener("ended", function() {
          document.body.classList.remove('videoplaying');
          });
        }
      };

    })( document, window );

    if (
        document.readyState === "complete" ||
        (document.readyState !== "loading" && !document.documentElement.doScroll)
    ) {
      window.mediatools();
    } else {
      document.addEventListener("DOMContentLoaded", window.mediatools);
    }

Maybe it is easier to sync the console preview and the presentation view when using a canvas element and to copy the content directly to the other window instead of propagating all possible events manually and triggering the corresponding actions? Is this possible? Which impact does this have on performance?

[henrikingo]

Thanks for sharing the code.

Using canvas would be a totally different presentation software. For the speaker console, the main architecture to understand is that the pre-view windows are essentially slaves to the main presentation window. So clicking on the navigation toolbar in the preview window doesn't really do us any good - assuming we stay with current architecture. However, simply rendering a new toolbar into the speaker console, that would be wired to control the main presentation, just like the current navigation buttons are, is indeed an interesting idea.

[complanar]

Okay, you’re right with canvas. But I’ve always wondered, why the console is a slave to the presentation and not the other way round. Nevertheless it should possible to create a “backlink” to the presentation window so that you can implement a bi-directional control. That does’t solve the toolbar problem, but enables media control by synchronizing playback.

I’ve added one line to the console’s open() method:

consoleWindow.impress = window.impress;
consoleWindow.presentationView = window; // <- new line

After that the following code should work out of the box without need of changing anything in the presentation file itself, except of including it somehow.

  (function (document, window) {
    "use strict";

    var consoleMediatools = window.consoleMediatools = function (rootId) {

      rootId = rootId || 'impress';

      // Root presentation elements
      var root = document.getElementById( rootId );

      var syncMedia = function (localObj, eventName) {
        var remoteDocument;
        var localId = localObj.getAttribute('id');
        if (top.isconsoleWindow) {
          remoteDocument = top.presentationView.document;
        } else {
          if (window.consoleWindow) {
            // Uncomment the following line to allow bi-directional controlling.
            // This unfortunately is buggy, because some kind of race conditions in the following 
            // lines that prevent infinite loops.

            //remoteDocument = window.consoleWindow.document.getElementById('slideView').contentDocument;
          }
        }

        // Try to control remote media object only if a remoteDocument (presentation or console) 
        // could be reached
        if (remoteDocument !== null && remoteDocument != undefined) {
          var remoteObj = remoteDocument.getElementById(localId);

          // Prevent infinite loops of remote contols
          if (!remoteObj.isMaster) {
            // Local object seems to be master. 
            // Set master flag to true and allow control of the remote object.
            // Mute presentation audio to prevent doubled output, which can sometimes sound crappy.
            // In general this could be the other way round, but:
            // Do not mute console audio, to allow easier volume changes.
            localObj.isMaster = true;
            switch (eventName) {
              case "play":
                remoteObj.currentTime = localObj.currentTime
                if (top.isconsoleWindow) {
                  localObj.muted = false;
                  remoteObj.muted = true;
                } else {
                  remoteObj.muted = false;
                  localObj.muted = true;
                }
                remoteObj.play();
                break;
              case "pause":
                localObj.muted = false;
                remoteObj.muted = false;
                remoteObj.pause();
                break;
              case "seeked":
                remoteObj.currentTime = localObj.currentTime;
                //localObj.play();
                //remoteObj.play();
                break;
              case "volumechange":
                remoteObj.volume = localObj.volume;
                if (top.isconsoleWindow) {
                  localObj.muted = false;
                  remoteObj.muted = true;
                } else {
                  remoteObj.muted = false;
                  localObj.muted = true;
                }
                break;
              case "ended":
                localObj.muted = false;
                remoteObj.muted = false;
                remoteDocument.body.classList.remove('videoplaying');
                remoteDocument.body.classList.remove('audioplaying');
                break;
              default:
                break;
            }
          } else {
            // Remote object seems to be master. 
            // Set master flag to false again and do not try to control anything.
            remoteObj.isMaster = false;
          }
        }
      }

      var initMediaSync = function (type) {
        var media = document.getElementsByTagName(type);
        for (var i = 0; i < media.length; i++) {
          var id = media[i].getAttribute('id');
          if (id === undefined || id === null) {
            media[i].setAttribute('id', type + i);
          }
          media[i].isMaster = false;
          media[i].addEventListener("play", function () {
            document.body.classList.add(type + "playing");
            syncMedia(this, "play");
          }, false);
          media[i].addEventListener("playing", function () {
            document.body.classList.add(type + "playing");
            syncMedia(this, "playing");
          }, false);
          media[i].addEventListener("pause", function () {
            document.body.classList.add(type + "paused");
            syncMedia(this, "pause");
          }, false);
          media[i].addEventListener("ended", function () {
            document.body.classList.remove(type + "playing");
            syncMedia(this, "ended");
          }, false);
          media[i].addEventListener("seeked", function () {
            syncMedia(this, "seeked");
          }, false);
          media[i].addEventListener("volumechange", function () {
            syncMedia(this, "volumechange");
          }, false);
        }
      }

      var steps = root.getElementsByClassName('step');
      for (var i = 0; i < steps.length; i++) {
        /* Pause audio and video automatically when leaving their step. */
        steps[i].addEventListener('impress:stepleave', function () {
          var videos = this.getElementsByTagName('video');
          for (var k = 0; k < videos.length; k++) {
            videos[k].pause();
            document.body.classList.remove('videoplaying');
          }
          var audios = this.getElementsByTagName('audio');
          for (var k = 0; k < audios.length; k++) {
            audios[k].pause();
            document.body.classList.remove('audioplaying');
          }
        });
      }

      initMediaSync('video');
      initMediaSync('audio');

     // New API for impress.js plugins is based on using events
      document.addEventListener( 'impress:console:open', function() {
        // Bind console window to the main window
        // Somehow does not work always. 'impress:console:open' event seems not to be fired always.
        window.consoleWindow = window.impressConsole = impressConsole().open();
      });
    };

    // This initializes consoleMediatools automatically when initializing impress itself
    document.addEventListener( 'impress:init', function( event ) {
      consoleMediatools();
    });
  })(document, window);

I struggled a bit with the difference of "playing" and "play" events of media, somehow the seem to be fired in different order, which makes catching and synchronizing them a bit tricky. This resulted in not restarting playback directly after a pause event. After clicking twice it worked again. Also seeking in between did restore the expected behaviour. Sadly, I was not able to dig deeper in. Didn’t completely understand the underlying reasons for this strange behaviour.

With a one way sync (console -> presentation) everything works fine, so I commented out the line that enables bi-directional control.

[complanar]

I’m currently working on making this a real plugin which meets the requirements defined in src/plugins/README.md. This will take a little time as I’m quite busy at the moment … I’ll create a pull request, when I’m ready.

[henrikingo]

Thanks. This looks cool.

There's no rush. I'd like to take some time to release a 1.0.0 version in the near future. This would in any case be merged only after that.

This thread now includes 2 different discussions. To take one step at a time, could you proceed as follows:

• Implement a plugin media and maybe a separate one for mediacontrols.
• Have the media controls integrate with the toolbar plugin, just like navigation-ui does. (Yes, this means they show up on the presentation side.)
• Separately address the issue of making these controls available in the speaker console. My suggestion is that impressConsole will trigger an event impress:console:opened, with a reference to the opened window (consoleWindow) in event.detail. Then toolbar should catch this and render a copy of the toolbar in the consoleWindow.
• Create separate issues, or in any case separate PRs, of each of the above.

[lwhshugyosha]

Hi, both,

just stumbling upon this discussion. Some weeks ago I already created a plugin playing media for my own use and was now checking how to release it to the general public. I'll better dig through this first and then come back.

[complanar]

I created a PR for a simple media plugin. See #672. It covers the following features:

• Autoplay and autopause on impress:stepenter and impress:stepleave
• Add a css class to the body tag when playing media (e.g. for darkening the background)
• Add some useful shorthands for inserting media (useful when making extensively use of html5 video and audio because it saves much typing). One could of course argue if this should even be a feature of impress.js or if this should be covered by some other (external) scripts (e.g. markdown integration).

Still working on turning the playback sync script into a real plugin (above code did it’s job, though).

I did not completely understand what you meant with the media controls integrate with the toolbar plugin. My solution was a more general approach. Just control the media with the help of normal html5 controls, catch the video/audio events in one window and propagate them to the corresponding nodes of the other window (console -> presentation and presentation -> console).

This should work independently of any other plugin (except impressConsole, of course) with an arbitrary number of media on each step and even If you’re using an external javascript library like e.g. video-js or plyr.

I could not imagine a use case where separate buttons on the toolbar should have any advantage, except for having invisible audio. Hiding media in the presentation view but not in the console window could be achieved with css, though, couldn’t it?

[lwhshugyosha]

That's about what I was doing, because I need it for a presentation in the near future.

Additionally, the plugin has a volume setting to level out media from different sources, and the ability to play a sound during the transition from the previous step. It can play the video inside the console as well, if so desired, but without sound to avoid echoes, or show a placeholder image instead.

Adding a css class while playing looks like a good idea.

Unfortunately, I wasn't successful with the PR so far...

[complanar]

You’re right. Muting one source is of course important. I did this in consoleMediatools.syncMedia(). I choose to mute the output of the presentation view, because that way it is easier to control the volume level within the speaker console with the regular controls. If you do it the other way round the volume slider behaves a little jerky, because most browsers and "html5-javascript-players" set it to zero automatically to indicate the the sound is completely muted.

One could of course show a place holder image, but I like to be able to see the video playback. This of course increases CPU load significantly. Theoretically the output could get out of sync after a while (e.g. due to network issues), but if you want to show a long HD video, you should maybe not use impress.js for that at all.

Being able to play sound during transitions sounds like a good idea. Personally I would not use that feature (not my taste), but if someone likes to catch the audience’s attention that way, why not. I find too much sound distracting from the real content. One could of course say the same about some transitions or 3D positioning, but It is not impress.js’ aim to force people into creating what we think is a good presentation. (That is a job for LaTeX ;-))

What do we need a separate mute button or volume slider for? If you want to play external sound during the presentation why not include it in the presentation? Or do you speak of showing a video and playing audio of a different source simultaneously. Why not combining the two with the help of ffmpeg or something like that?

[lwhshugyosha]

Well, it works a bit different. Sound control is just another attribute for the media tag (e.g. data-volume="0.5"), so that if You play several samples during Your presentation, You can pre-adjust them to a common sound level. No control buttons or such. I'd prefer to play the sound from the main screen, because that is what the audience sees and should hear as well - in case the two playbacks get out of sync.

For my current tasks I dont't use the HTML5 playback controls at all.

But why haggle? In the final version we can add atrributes for every variant ;-)

[henrikingo]

I did not completely understand what you meant with the media controls integrate with the toolbar plugin. My solution was a more general approach. Just control the media with the help of normal html5 controls, catch the video/audio events in one window and propagate them to the corresponding nodes of the other window (console -> presentation and presentation -> console).

Ok, that makes sense I guess. I guess that's what you said already, but in this case I'd still want to approach this with a more general solution. If you want media objects in the impressConsole window to be clickable, then why not the toolbar, links, etc... too? So basically anything that happens in the impressConsole iframe should be propagated to the main presentation window.

At the same time, the main window remains the main window and opening impressConsole is optional. Therefore it should in any case remain possible to control everything from main window, and events in main window must propagate to the impressConsole preview windows.

I'm not sure having such bi-directional propagation of control and events is easy. (How do you prevent infinite loops?) But that's in any case my current vision for what the requirements are.

[henrikingo]

Being able to play sound during transitions sounds like a good idea. Personally I would not use that feature (not my taste), but if someone likes to catch the audience’s attention that way, why not. I find too much sound distracting from the real content.

I think a plugin to play sound effects tied to transitions is something I'm sure will happen sooner or later. Sure, it will probably be annoying, but nobody is forced to use that.

I would expect that to be a different plugin compared to the media plugin discussed here. (One plays media on a slide, the other related to transitions.)

[janishutz]

Will be looking into this