search

AILab-CVC / FreeNoise

[ICLR 2024] Code for FreeNoise based on VideoCrafter
http://haonanqiu.com/projects/FreeNoise.html
Apache License 2.0
389 stars 24 forks source link

Add Replicate demo and API #8

Closed chenxwh closed 1 year ago

chenxwh commented 1 year ago

Hey @arthur-qiu!

This pull request makes it possible to run LongerCrafter on Replicate (https://replicate.com/cjwbw/LongerCrafter) and via API (https://replicate.com/cjwbw/LongerCrafter/api).

I did have a question regarding the output size, do they need to be either 576x1024 or 256x256? Tested other settings had either error or low quality, think this might be a good update :)

Happy to transfer the ownership of the page to your account so you can easily push any changes!

arthur-qiu commented 1 year ago

Hey @arthur-qiu!

This pull request makes it possible to run LongerCrafter on Replicate (https://replicate.com/cjwbw/LongerCrafter) and via API (https://replicate.com/cjwbw/LongerCrafter/api).

I did have a question regarding the output size, do they need to be either 576x1024 or 256x256? Tested other settings had either error or low quality, think this might be a good update :)

Happy to transfer the ownership of the page to your account so you can easily push any changes!

Many thanks for your contribution!

Here we use the VideoCrafter as our base model and it mainly supports the fixed resolution. Similar resolutions can be generated by changing the image_size and scale_b in the config correspondingly. In this way, quality will degrade based on the gap between training and inference. We will recover the model of 376 x 512 resolution soon. In addition, FreeNoise + ScaleCrafter (video version) should be a good solution for the flexible resolution and we plan to integrate them in the future.

Btw, can you replace the example of "a man is skiing in the snow"? We find the 1024 model is not good on motion.

Finally, you can host the Ownership of Replicate until you don't want to maintain it. Thank you again for your contribution.