Flatten stack produces undesirable results if track does not end with gap or clip

[douglascomet]

Bug Report

Repro test

C++:

// SPDX-License-Identifier: Apache-2.0
// Copyright Contributors to the OpenTimelineIO project

#include "opentime/timeRange.h"
#include "utils.h"

#include <opentimelineio/clip.h>
#include <opentimelineio/stack.h>
#include <opentimelineio/track.h>
#include <opentimelineio/stackAlgorithm.h>

#include <iostream>

namespace otime = opentime::OPENTIME_VERSION;
namespace otio  = opentimelineio::OPENTIMELINEIO_VERSION;

int
main(int argc, char** argv)
{
    Tests tests;

    tests.add_test(
        "test_flatten_stack", [] {
        using namespace otio;

        otio::RationalTime rt_0_24{0, 24};
        otio::RationalTime rt_150_24{150, 24};
        otio::TimeRange tr_0_150_24{rt_0_24, rt_150_24};

        // all three clips are identical, but placed such that A is over B and
        // has no gap or end over C
        // 0         150          300
        // [    A     ]
        // [    B     |     C     ]
        // 
        // should flatten to:
        // [    A     |     C     ]

        otio::SerializableObject::Retainer<otio::Clip> cl_A =
            new otio::Clip("track1_A", nullptr, tr_0_150_24);
        otio::SerializableObject::Retainer<otio::Clip> cl_B =
            new otio::Clip("track1_B", nullptr, tr_0_150_24);
        otio::SerializableObject::Retainer<otio::Clip> cl_C =
            new otio::Clip("track1_C", nullptr, tr_0_150_24);

        otio::SerializableObject::Retainer<otio::Track> tr_over =
            new otio::Track();
        tr_over->append_child(cl_A);

        otio::SerializableObject::Retainer<otio::Track> tr_under =
            new otio::Track();
        tr_under->append_child(cl_B);
        tr_under->append_child(cl_C);

        otio::SerializableObject::Retainer<otio::Stack> st =
            new otio::Stack();
        st->append_child(tr_under);
        st->append_child(tr_over);

        auto result = flatten_stack(st);

        assertEqual(result->children()[0]->name(), std::string("track1_A"));
        assertEqual(result->children().size(), 2);
        assertEqual(result->duration().value(), 300);
    });

    tests.run(argc, argv);
    return 0;
}

Python:

import opentimelineio as otio

rt_0_24 = otio.opentime.RationalTime(0, 24)
rt_150_24 = otio.opentime.RationalTime(150, 24)
tr_0_150_24 = otio.opentime.TimeRange(
    start_time=rt_0_24,
    duration=rt_150_24
)

# all three clips are identical, but placed such that A is over B and has no
# gap or end over C
# 0         150          300
# [    A     ]
# [    B     |     C     ]
# 
# should flatten to:
# [    A     |     C     ]

clip_A = otio.schema.Clip(name="track1_A", source_range=tr_0_150_24)
clip_B = otio.schema.Clip(name="track2_B", source_range=tr_0_150_24)
clip_C = otio.schema.Clip(name="track2_C", source_range=tr_0_150_24)

tr_over = otio.schema.Track(children=[clip_A])
tr_under = otio.schema.Track(children=[clip_B, clip_C])

st = otio.schema.Stack(children=[tr_under, tr_over])

result = otio.algorithms.flatten_stack(st)

# does select A on top for the first clip
assert(result[0].name == 'track1_A')

# ...but trims C instead of including it
# the resulting track is just this:
# [    A     ]
assert(result[1].name == 'track2_C')

# these assertions also fail (see below)

# should have two clips
assert(len(result) == 2)

# with duration 300
assert(result.duration().value == 300)

If the stack is flipped (such that the track with two clips is above the track with one, then it functions as expected.

(original bug report follows below)

Incorrect Functionality and General Questions

When using flatten_stack if a video track does not end with a clip or gap above another video track that does the child from the bottom video track will be omitted from the returned timeline.

To Reproduce

1. Win 10
2. Python 3.9
3. OpenTimelineIO 0.14.1

Expected Behavior

flatten_stack should insert a gap prior to flattening or account for this scenario when flattening so the clip on the bottom track is accounted for and included in the returned timeline

Screenshots

Initial results, the top image from otioview is the original timeline and the bottom image from otioview is the result of flatten_stack

Tested this by inserting a new clip into the above track and re-ran flatten_stack and got the same result where the end of the bottom clip is being omitted from the returned timeline.

[meshula]

Thanks for the report!

[ssteinbach]

I added a simple test script to the bug report

[douglascomet]

fwiw, I was able to get around this issue by adding gaps programatically and skipping empty tracks

# Filter empty tracks because
# if the top track is empty then it will create an empty track after flattening
video_tracks = list(filter(None, timeline.video_tracks()))

rate = timeline.global_start_time.rate
end_frame = timeline.duration().to_frames() - 1
end_time = otio.opentime.from_frames(end_frame, rate)

# Iterate through video tracks and determine which do not end with a child and append a gap
for index, track in enumerate(video_tracks):

    result = track.child_at_time(end_time)
    if result:
        msg = 'Skipping adding gap to the track because' \
                ' it has a child at end time of the timeline.'
        print(msg)
        continue

    # Get time range of last child in track
    # Tracks are essentially lists of children reading left to right based on their
    # sequential order in the track
    source_range = track.range_of_child(track.children_if()[-1])
    print('Track {}\'s last child\'s source_range {}'.format(index, source_range))

    # The gap will be inserted after the last child's duration,
    # so calculate that frame by adding the last child's start frame and its duration
    gap_start_frame = source_range.start_time.to_frames() + source_range.duration.to_frames()
    print('Start frame for the gap to be created {}'.format(gap_start_frame))

    # Determine time range from start and end frame for the gap
    duration = end_frame - gap_start_frame + 1
    start_time = otio.opentime.RationalTime(gap_start_frame, rate)
    duration_time = otio.opentime.RationalTime(duration, rate)
    gap_source_range = otio.opentime.TimeRange(start_time=start_time, duration=duration_time)
    print('Time Range of gap: {}'.format(gap_source_range))

    # Create gap and append to the track
    gap = otio.schema.Gap(source_range=gap_source_range)
    track.append(gap)

one_track = otio.algorithms.flatten_stack(video_tracks)

[vfxbyjohn]

Came across this problem myself, where a "lower" track that was the last clip on the timeline and was missing from the result of otio.core.flatten_stack, but your workaround @douglascomet did work for me, thank you!

[jminor]

Fixed by https://github.com/AcademySoftwareFoundation/OpenTimelineIO/pull/1703

@douglascomet can you double-check that this fix works for your case?