search

seervideodiffusion / SeerVideoLDM

[ICLR 2024] Seer: Language Instructed Video Prediction with Latent Diffusion Models
16 stars 2 forks source link

Data preparation #1

Open abdo-RG opened 4 months ago

abdo-RG commented 4 months ago

Can you provide us how you preprocessed the datasets please? I downloaded bridge dataset from the official website with ~400GB but it's so complicated in comparison with the structure you mentioned in README!

XianfanGu commented 3 months ago

Hi, sorry for the late reply. It would be easy to preprocess the mp4-only data if you download the Bridgedata V1 dataset on [http://rail.eecs.berkeley.edu/datasets/bridge_dataset/demo_data_mp4.tar]. Please follow the instructions of mmaction2 code base (similar to Something-Something-v2) on [https://github.com/open-mmlab/mmaction2/blob/main/tools/data/sthv2/README.md], you only need to change the level of the saved directory in the script extract_rgb_frames_opencv.sh

Patricia1019 commented 2 months ago

The http://rail.eecs.berkeley.edu/datasets/bridge_dataset/demo_data_mp4.tar you provided has been invalid. I have downloaded the latest bridge data v2, and the overall data structure is totally different from yours. Can you provide the detail on how you preprocess the data?

XianfanGu commented 2 months ago

We found that http://rail.eecs.berkeley.edu/datasets/bridge_dataset/demo_data_mp4.tar is not available since this month. And sorry we don't include the dataloader for bridge data v2. However, our dataloader is quite simple (we only need extracted video frames and text annotation for each video). And I know that bridgedata v2 has already extracted video frames then you can simply modify the dataloader for processing bridgedata v2. For example, in the path 'raw/bridge_data_v1/berkeley/toykitchen1/close_microwave/2021-12-02_12-14-59/raw/traj_group0/traj1/images0' of bridgedata v2, the text prompt is 'close microwave' extracted from path, the video frames can be sampled from one of the trajectories (traj1~traj26). Hope this answer can solve your concern