PKU-YuanGroup / Open-Sora-Plan

This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.
Apache License 2.0
10.87k stars 971 forks source link

Support Video to Video generation #243

Open alfredplpl opened 2 months ago

alfredplpl commented 2 months ago

I would like to v2v by your model. I think we need to add two points on opensora/sample/pipeline_videogen.py.

  1. Create the encode_videos function like the follow:

    def encode_videos(self, videos):
        latents = self.vae.encode(videos)
        ...
        return latents
  2. Add code on the prepare_latents function so that the function copes with noised latents.

Any idea?

LinB203 commented 2 months ago

Very creative idea. We can make further attempts. But I believe that before that, our model needs to be generalised enough, which means a lot of data to train.