Syncing playback of multiple videos with arbitrary in / out times to a TimingObject

[stuburger]

Hi there! I've been playing around with your library this morning to see if it'd help me solve a problem I'm facing. It may be that this is not what this suite of packages is built for, but its the closest thing I could find! Do let me know if I'm barking up the wrong tree 😄

From what I can tell from your docs, timingsrc is only meant to sync the currentTime of two or more videos.

Challenge

I have a series of clips that form a sequence that should be playable as a continuous video. This might be familiar to you if you've ever done video editing using a tool like premier pro, for example.

const sequence = [
  { inTime: 5, outTime: 10, startTime: 0, endTime: 5, url: 'video1.mp4'  }, // clip 1
  { inTime: 60, outTime: 70, startTime: 5, endTime: 15, url: 'video2.mp4'  }, // clip 2
  { inTime: 700, outTime: 750, startTime: 20, endTime: 70, url: 'video1.mp4'  }, // clip 3
  { inTime: 0, outTime: 20, startTime: 60, endTime: 80, url: 'vide3.mp4'  }, // clip 4
]

1. Each clip in a sequence has an inTime and outTime which represent the parts of the original media (the url property) that I'd like to play.
2. startTime and endTime are "cues" in the "master" timing object where I'd like to start and stop playback of a particular clip.
3. Clips may or may not have silence in between them (see transition from clip 2 -> clip 3)
4. Clips may overlap when a clip's startTime is before the endTime of another clip. (clip 4 and clip 3 overlap for 10s)

Currently I am not using timingsrc but only the TimingObject. A rough implementation looks something like this:

const sequence = [...]
const timingObject = new TimingObject()

let current = null

requestAnimationFrame(function update() {
  const vector = timingObject.query()

   const clipToPlay = sequence.find((clip) => {
      return (
        vector.timestamp >= clip.startTime &&
        vector.timestamp < clip.endTime
      )
    })

  if(clipToPlay !== current) {
      if (current) {
        current.videoElem.pause();
      }

      current = clipToPlay;
      current.videoElem.currentTime = source.inTime;
      current.videoElem.play();
  }

  requestAnimationFrame(update)
})

The above is obviously seriously flawed because clip playback does not correct itself back to the timing object, playback ends up drifting and the sequence looks and sounds nothing like it should.

Question

Is what I am attempting to do possible using timingsrc? Is this even the kind of problem is was built to solve? Any suggestions or advice are welcome!

[chrisguttandin]

Hi @stuburger, thanks for giving this package a try.

From what I can tell from your docs, timingsrc is only meant to sync the currentTime of two or more videos.

I would say it's meant to sync the currentTime of a media element (which could also be audio) with a TimingObject. To synchronize two or more media elements one can re-use the same TimingObject for all those media elements.

I think you could use five TimingObjects instead of one.

I would suggest to use one TimingObject to model your main timeline (from 0 to 80). Let's call it mainTimingObject. And four more for each video which depend on the mainTimingObject. The first one could look like this:

const firstVideoTimingObject = {
    query () {
        const mainTimingVector = mainTimingObject.query();

        return { ...mainTimingVector, position: Math.min(Math.max(0, mainTimingVector.position + 5), 5)  };
    }
    update (/*...*/) {
        // ... adjust the vector accordingly ...
    }
    // ... maybe just relay all other properties of the mainTimingObject ...
}

The idea is that you map the current position of the main timeline to the position of each individual video.

I think this is a common pattern. Maybe it makes sense to implement a generic TimingObject which just takes another TimingObject and a mapping function to make this a bit easier.

[stuburger]

Interesting, I would not have considered this approach but I see how this may work. Thank you for your insight! I'll give this a try and report back.

[chrisguttandin]

Hi @stuburger, did you have a chance to give this a try? Did it work out?

[chrisguttandin]

There is actually a third parameter that can be used to modify the vector before it gets applied. It was undocumented until now which is why even I forgot about it. 🤦

This is the example for the first video from above using the existing functionality.

setTimingsrc(
    mediaElement,
    timingObject,
    ({ position, ...vector }) => ({ ...vector, position: Math.min(Math.max(0, mainTimingVector.position + 5), 5) })
);

I'll add a section to the readme.