search

PKU-YuanGroup / Open-Sora-Plan

This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.
Apache License 2.0
10.88k stars 972 forks source link

The performance gap between using vanilla 3dVAE and Casual3dCNN #236

Open Sutongtong233 opened 2 months ago

Sutongtong233 commented 2 months ago

Hi, I am interested in the performance of using 3d CNN without casual for VAE training, and corresponding results of generation. As I understand, the propose of Casual3dCNN is for autoregressive model of stage two in MagViT. Since our stage two is still diffusion, is there any consideration of using Casual3dCNN?

LinB203 commented 2 months ago

We are currently just inflating 2DVAE to 3DVAE and replacing conv3d with causalconv3d. this already supports simultaneous training of images and videos.