Multicam Support

[eric-with-a-c]

Are there any plans for adding a schemadef for multicam clips, or is the preferred direction to just use a stack to represent multicam clips?

The downside to using a stack is that if you round trip through OTIO back to an NLE you would lose the multicam clips and they would just become nested tracks.

[jminor]

Yes, we would like to add a schema for this, but we don't have it on the schedule. Would you like to propose a schema for it?

[ssteinbach]

Just to be clear, is the assumption with a multicam item that:

• each 'take' referenced by the multiclip is the same duration and has the same start time?
• that each 'take' referenced by the multiclip is of the same 'kind'? (they're all Video or all Audio)
• there is one 'primary' or 'pick' or 'select' that can be used by the adapter if it doesn't support multicam clips

Could this also be used for stereo pairs?

We've discussed two different ways this could be expressed:

• As a media reference with multiple targets
• as a composition with multiple clips in it

Definitely open to suggestion on this if you have any ideas about what makes sense to you!

[eric-with-a-c]

@ssteinbach before I jump in with answers to things you may not be asking, when you refer to a 'take' are you referring to each piece of media that's included inside the multicam clip?

[ssteinbach]

@ssteinbach before I jump in with answers to things you may not be asking, when you refer to a 'take' are you referring to each piece of media that's included inside the multicam clip?

Yup! Sorry about that. Thats exactly what I meant.

[jminor]

Incidentally, I was just looking at how AAF handles this via their concept of a Selector. It is an object that has one or more "alternates" of which, zero or one is "selected". In the OTIO schema, this would be a subclass of Item, and a peer with Clip. We should compare this with how FCPXML models a multicam clip. Is it more like a sub-composition, or a special kind of clip?

[eric-with-a-c]

For multicams in FCP X there are two pieces. The first is the multicam clip itself which is a container for several pieces of media, both audio and video, each one being referred to as an angle (even if it's just audio). Each piece of media that make an angle typically have a different start and end time. The angles are synced/aligned using either timecode or audio. For syncing to work using audio there must be scratch audio in the video files.

This is what the clip looks like in the XML:

<media id="r9" name="Untitled Multicam Clip" uid="AjO7aKUbSFGPA/UPveBtQA" modDate="2019-02-21 18:27:35 -0500">
    <multicam format="r2" tcStart="0s" tcFormat="NDF">
        <mc-angle name="IMG_0023" angleID="5TZ2uW/AT16v4hs32EW++g">
            <gap name="Gap" offset="0s" duration="6100/3000s" start="3600s"/>
            <asset-clip name="IMG_0023" offset="6100/3000s" ref="r3" duration="30400/3000s" audioRole="dialogue" tcFormat="NDF"/>
        </mc-angle>
        <mc-angle name="IMG_0233" angleID="0ZqhPpuoSiWU1Z0lXyOHbQ">
            <asset-clip name="IMG_0233" offset="0s" ref="r4" duration="135600/3000s" audioRole="dialogue" tcFormat="NDF"/>
        </mc-angle>
        <mc-angle name="IMG_0268" angleID="ctboIx5cRdKIGi8EEDDYxQ">
            <gap name="Gap" offset="0s" duration="3200/3000s" start="3600s"/>
            <asset-clip name="IMG_0268" offset="3200/3000s" ref="r5" duration="110600/3000s" audioRole="dialogue" tcFormat="NDF"/>
        </mc-angle>
        <mc-angle name="IMG_0513" angleID="erFvoohVTVe2JSOXweZCaA">
            <gap name="Gap" offset="0s" duration="9800/3000s" start="3600s"/>
            <asset-clip name="IMG_0513" offset="9800/3000s" ref="r6" duration="31200/3000s" audioRole="dialogue" tcFormat="NDF"/>
        </mc-angle>
    </multicam>
</media>

The second part is the actual use of the multicam clip. So if you put the multicam clip on the timeline, and play through it, you can cut between the different angles. Each of these cuts is represented in the XML by referring to the multicam clip and identifying which angle should be used, what its start time is, what its offset is, and if the audio, the video, or both should be used from that angle.

<project name="Untitled Project" uid="2A71E7F4-5F65-4F7C-86CE-F65A0E915CF3" modDate="2019-02-21 18:27:35 -0500">
    <sequence duration="135600/3000s" format="r2" tcStart="0s" tcFormat="NDF" audioLayout="stereo" audioRate="48k">
        <spine>
            <mc-clip name="Untitled Multicam Clip" offset="0s" ref="r9" duration="3300/3000s">
                <mc-source angleID="5TZ2uW/AT16v4hs32EW++g" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="3300/3000s" ref="r9" duration="12400/3000s" start="3300/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="15700/3000s" ref="r9" duration="10400/3000s" start="15700/3000s">
                <mc-source angleID="erFvoohVTVe2JSOXweZCaA" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="26100/3000s" ref="r9" duration="5800/3000s" start="26100/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="31900/3000s" ref="r9" duration="8400/3000s" start="31900/3000s">
                <mc-source angleID="ctboIx5cRdKIGi8EEDDYxQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="40300/3000s" ref="r9" duration="10400/3000s" start="40300/3000s">
                <mc-source angleID="ctboIx5cRdKIGi8EEDDYxQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="50700/3000s" ref="r9" duration="9100/3000s" start="50700/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="59800/3000s" ref="r9" duration="6300/3000s" start="59800/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="66100/3000s" ref="r9" duration="19700/3000s" start="66100/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="audio"/>
                <mc-source angleID="ctboIx5cRdKIGi8EEDDYxQ" srcEnable="video"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="85800/3000s" ref="r9" duration="9500/3000s" start="85800/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="95300/3000s" ref="r9" duration="7900/3000s" start="95300/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="audio"/>
                <mc-source angleID="ctboIx5cRdKIGi8EEDDYxQ" srcEnable="video"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="103200/3000s" ref="r9" duration="8300/3000s" start="103200/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="all"/>
            </mc-clip>
            <mc-clip name="Untitled Multicam Clip" offset="111500/3000s" ref="r9" duration="24100/3000s" start="111500/3000s">
                <mc-source angleID="0ZqhPpuoSiWU1Z0lXyOHbQ" srcEnable="audio"/>
                <mc-source angleID="ctboIx5cRdKIGi8EEDDYxQ" srcEnable="video"/>
            </mc-clip>
        </spine>
    </sequence>
</project>

The start in the mc-clip is relative to the multicam clip, not the media that makes up the angle that's being referred to, if that makes sense.

Maybe this is a long of saying that multicam clips are both a special kind of clip and a kind of composition. It seems like we would need a special kind of media reference in OTIO that could group all the different angles together, with some kind of offset or gap to align them with each other, and a way of referring to a particular angle within that media reference from an OTIO Clip.

In case it's not obvious, I haven't thought too much about this and am just spitballing. 😂

[ssteinbach]

@eric-with-a-c when you say "So if you put the multicam clip on the timeline, and play through it, you can cut between the different angles." do you mean that the "active" clip is an animated parameter, or that the multicam clip behaves kind of like a nested track with cuts of clips inside of it?

Is the tcStart a time code in which the clip starts playing and the previous stops? Can you have multiple things playing at once?

[eric-with-a-c]

I would say that it's more like a nested track with cuts applied to it. That act of cutting between angles is very similar to switching between feeds of a live broadcast.

Yes, you could have multiple things playing at once. For example, you might be playing the video from one angle but the audio from another.

Just for posterity, here are the links to the Final Cut Pro X and Adobe Premiere Pro documentation on multicam editing. https://support.apple.com/kb/PH12656 https://helpx.adobe.com/premiere-pro/using/create-multi-camera-source-sequence.html

Davinci Resolve also supports multicam, but all their documentation is in PDF format so you have to download it from their support site. The NLE I'm least familiar with is Media Composer (sorry), but I think they have a way of doing multicam work too, I just don't have access to any of that documentation.

[virokannas]

One way to visualize how FCXP handles multicam is to think of it as a separate subtimeline, and what gets presented is a "window" that has a start and end, and between them both the video and audio playheads might jump between tracks in that subtimeline. This jumping is the animated parameter, which gets presented as a "cut" but it's not a hard cut from a clip to another as far as the main timeline is concerned.

One way to present this at the top level is to flatten multicam clips and just separating these "angle cuts" into individual clips. That process, however, is not reversible.

As @eric-with-a-c mentioned, there can be up to two things (video and audio) playing at once inside the multicam clip. Not sure if that's the same for 5.1 audio or 3D or other things like that.

[jminor]

Thanks, this is super helpful. It sounds to me like this needs to be an OTIO Composition - like Stack and Track, which can be nested within each other, rather than just a special Clip. That will allow any sort of sub-composition inside. As to how we encode the animated cuts... we'll have to think about that a bit.

I'd like to learn more about how AAF does this (and any other relevant format that OTIO supports) so we can find a nice common ground that will facilitate interchange (if possible).

[virokannas]

Just to elaborate a little bit on the previous comment from over a year ago: the animated parameter would need to be separate for the audio and video components to fully support what FCPX is doing. I would imagine at this point that is the most complicated scenario it would need to accomplish.

To be able to round-trip back to fcpxml from OTIO, if it's an OTIO Composition, it would still need the necessary metadata to imply it's a multicam clip so that it can be reconstructed as one.

[russellaugust]

As far as I can tell (and from what Adobe has told me), Adobe does not support multicam in it's XMLs so I don't think there will be a way in it's current form to get multicam edits from Premiere -> OTIO. Even Extendscript doesn't truly support multicam.

[virokannas]

@russellaugust That sounds like a very API-specific limitation for Premiere. It does have multicam support - is this something that could be accessed using a binary plug-in? It would still be nice if OTIO had a schema for storing this information. There will probably always be specific things some NLEs can't accommodate, but in such cases even having a one-way mapping would at least enable working through OTIO.

To do a one-way mapping the multicam clips (both video and audio) could turn that animated parameter into in/out points for the underlying media clips and kick them out to a "flattened" track. If the original multicam info could be stuffed into a metadata field of some sort, at least the parts that weren't modified could be mapped back as-is?

I don't know how Avid handles multicam timelines, maybe someone has more insight on that? It would be great if the schema would at least accommodate all current multicam-capable NLEs. @eric-with-a-c already provided an FCPX example, maybe something similar could be posted for Media Composer?

[russellaugust]

@simpassi

Skimming through the C++ SDK, there appears to be no support for manipulating the actual timeline. Just reading and writing to the project bins and effecting the playback itself (video filters, screen overlays, etc).

Extendscript has the ability to manipulate the timeline but has no available access to the multiclips camera angles. It sees them as nested sequences. It's also very slow to work with.

XML export deletes multicam clips before exporting.

AAF export flattens the multicams before exporting.

EDIT: Sorry, I didn't clarify that I'm still talking about XML and plugin support from Premiere. Premiere does not have a front-facing way of identifying what multiclip camera is active.

[ssteinbach]

@russellaugust After reading in the timeline, you can manipulate the objects in memory and write them back out, using their fields and methods... unless I'm misunderstanding what you're suggesting? For example: https://opentimelineio.readthedocs.io/en/latest/cxx/cxx.html#using-schemas

That example adds some clips to a track.

[russellaugust]

@ssteinbach apologies, i was referring to Premiere's and how it has no visible way to export multiclip data, even through SDK plugins.

[ssteinbach]

Ah sorry misunderstood! Carry on!

[TrevorAyl]

As an Avid user (not a coder) I can tell you how it appears to an editor -

You can 'group' clips and 'multi-group' clips.

'Multi-group' is a group of groups - used on stop / start multi-cam shows with time of day timecode - so all the clips can be grouped together and edited into a timeline - the first match frame will take you to the 'Whole Day' multigroup, match-frame there to get to the individual 'Take' group - then match-frame back to the master (or sub-clip or sync-clip or ??) that the group was made from.

Groups/MultiGroups have audio from the source clips as mono tracks.

With a group clip loaded in the source monitor the top left icon allows the choice of which of the angles is 'selected' for the clip - and there is an 'audio-follow-video' choice here.

Once cut in a timeline they are separately switchable (i.e. There's no audio follow video option here).

Group clips can only be mono sources. Any clips that are set to be anything other than mono are set to mono inside the group clips. (Which is often an annoying limitation.)

They very much appear as timelines - more recent versions have allowed 'editing' of groups - e.g. to resync a camera - that open in the timeline in Avid for changes to be made.

In the group timeline all audio tracks for a clip (with audio) are represented by 1 audio track - but all tracks are available when using the group.

Not sure how robust mixed frame rates are in a group (this is just a bunch of random clips in a 50P project). -For clarity the image below is whilst 'editing' a group (i.e. not a regular timeline):

In the AAF world, as mentioned above, they use a 'Selector' class - with one compulsory 'Selected' angle and other 'Alternates'. Have never managed to create an AAF like this successfully - so if anyone does have the example code for it ...(including how it might need to be in it's own CompMob?).

EVS have a product (Directors Cut - part of IP Director) that will take switcher tallies and generate a timeline of corresponding grouped clips linked to EVS files (AAF for Avid, XML for Apple, Adobe). Haven't seen it in the wild ever but it was at IBC a few years ago and is still showing as a product.

Hope that helps.