curating high-quality video data

[pengzhangzhi]

Hi team members, I would attribute the success of SORA to the training data like how OpenAI has done for GPT. Any ideas on curating high-quality video data?

[pengzhangzhi]

A random thought. As a community, would be great to have a discord channel for discussions and updates.

[bhack]

You could also check https://laion.ai/blog/video2dataset/

[aluo-x]

I should be able to help on the high quality video data aspect (along with transcripts), although the source of captions is a more difficult problem.

[cxh0519]

Our preliminary goal is to achieve impressive results in a specific data domain to verify the effectiveness of our pipeline and then extend our plan for generalization. Please keep attention on our project and feel free to contact us for discussion and potential cooperation.

[sayakpaul]

Is the domain of the data fixed? If so, do you have more public information available on that?

Regardless, I think having some sort of a data curation pipeline (like the one used in the Stable Video Diffusion paper) would be really nice.

[cxh0519]

Agree. After the preliminary validation, we will construct a data curation pipeline following successful projects and pay more attention to data.

[sayakpaul]

Cool, sounds like a plan. FWIW, I put together a simple repository that walks through the primary steps of the Stable Video Diffusion curation pipeline: https://github.com/sayakpaul/single-video-curation-svd.

[cxh0519]

We will check it and thanks for your effort!

[yuxuan-li92]

You could also check data-juicer, it seems to be beneficial for video data curation.

[bhack]

https://arxiv.org/abs/2403.06098