Support for segmented video inputs

[MaxPower15]

My team and I operate a parallel rendering system, similar to Remotion in some ways, that segments input media before rendering, then downloads the minimum required files to each worker that's responsible for a frame range. This drastically reduces overall time spent for each worker when operating on large input files. I am seeking to use Remotion in this system.

I was immediately drawn to the frameRange render option (--frames on the CLI) to accomplish this since I believe this is what's used in lambda rendering. However, Remotion's assumption that the full input file exists locally--which is not desired in my case--made it impossible (?) to make it work without changing something in Remotion itself.

I made an example repo here that demonstrates the issue: https://github.com/remotion-dev/remotion/compare/main...wistia:remotion:mms/segmented-video/example-issue. You can run the commands below to reproduce. You should see an output video, but everything after the first 3 seconds is just black frames.

git checkout mms/segmented-video/example-issue
cd packages/examples
mkdir -p out
pnpm exec remotion render segmented-video-example --frames 0-89 --output out/big-buck-bunny-output-seg1.mp4
pnpm exec remotion render segmented-video-example --frames 90-179 --output out/big-buck-bunny-output-seg2.mp4
pnpm exec remotion render segmented-video-example --frames 180-209 --output out/big-buck-bunny-output-seg3.mp4
echo "file 'big-buck-bunny-output-seg1.mp4'
file 'big-buck-bunny-output-seg2.mp4'
file 'big-buck-bunny-output-seg3.mp4'" > out/concat.txt
ffmpeg -y -hide_banner -f concat -i out/concat.txt -c copy out/big-buck-bunny-output.mp4
ffplay out/big-buck-bunny-output.mp4

Feature Request 🛍️

I would like some way to make segmented input media work. There may be several approaches to this, but that's the gist of it.

Use Case

Specifically, I'd like to be able to tell each media component (i.e. Video, OffthreadVideo, Audio) "I am managing your offset myself with startFrom and endAt--don't worry what the frame range says."

Possible Solution

I implemented a solution for OffthreadVideo--an ignoreFrameRange prop--as an example here: https://github.com/wistia/remotion/compare/mms/segmented-video/example-issue...wistia:remotion:mms/segmented-video/example-issue-fixed.

If you run the same commands as above, but on the mms/segmented-video/example-issue-fixed branch, it should produce the appropriate output.

The idea with this change is basically, "let the user opt out a media component from being constrained by the frameRange option." By doing that, I can use my external logic and calculations to do the same job.

That said, if there is a better way to accomplish this, I'm all ears and would be happy to take it on! I just did what seemed to be the minimal required Remotion changes to accomplish my goal in a somewhat sane way. But if this seems like a good idea, I'd be happy to flesh it out more and contribute a PR. Let me know!

EDIT: I have a couple other ideas for this that don't involve changing Remotion. I'll report back if they work.

[MaxPower15]

Seems obvious now, but I came up with a seemingly better solution that just enumerates all the given segments using the Sequence component instead of the function that changes props for a single one. I'll close this as that seems better!

The example above using that solution: https://github.com/remotion-dev/remotion/compare/main...wistia:remotion:mms/segmented-video/alt-solution-no-change

[JonnyBurger]

Makes sense!

Since this is a custom use case I also think it would be better if it was implemented just by passing this information via input props. Seems doable to me.

For our "native" solution, we want to natively only download the portion of the video that is needed. Happy to tackle this in the future and conversate with you about this 🙂