search

jjihwan / FIFO-Diffusion_public

Official implementation of FIFO-Diffusion: Generating Infinite Videos from Text without Training (NeurIPS 2024)
https://jjihwan.github.io
397 stars 26 forks source link

Dissapointing sora results #14

Closed SoftologyPro closed 5 months ago

SoftologyPro commented 5 months ago

I am trying the new open-sora support. The origin.mp4 results look OK. The final fifo_vae versions look like they have been put through a bad interpolator (they have that "jelly wobble" look moviers get after being processed with FILM, DAIN, RIFE etc) rather than the smooth results the original FIFO-Difssuion created, Is this a known issue that will be fixed, or is that the best sora will do? Also works fine on a 24GB 4090, but takes twice as long as VideoCrafter2.

Here is the shortest sample I could squeeze into githubs 10mb limit https://github.com/jjihwan/FIFO-Diffusion_public/assets/47686889/35ee2df9-eece-434f-ac5c-18827b276d48 Notice how the chicken seems to deform and wobble. The other samples have the same result.

jjihwan commented 5 months ago

It seems to be due to the VAE of the Open-Sora Plan, and I think it cannot be resolved easily. I'll try to check it soon.

SoftologyPro commented 5 months ago

Thanks. I have gotten some really nice results from the VideoCrafter2 models. https://www.instagram.com/softology/

jjihwan commented 5 months ago

It seems great!!