search

OpenDriveLab / Vista

[NeurIPS 2024] A Generalizable World Model for Autonomous Driving
https://vista-demo.github.io
Apache License 2.0
497 stars 32 forks source link

Data on nuscenes #5

Closed lianglia-apollo closed 2 months ago

lianglia-apollo commented 3 months ago

First, appreciate for open source this project.

Could you elaborate which dataset is needed on https://www.nuscenes.org/download to reproduce this work?

Little-Podi commented 3 months ago

Hi. Please download all splits of Trainval in Full dataset (v1.0), including the sweeps set. image To replicate our performance, we recommend also downloading OpenDV-YouTube dataset from DriveAGI if possible. You can start with a few YouTubers like J Utah. Feel free to let us know if you encounter any troubles.

MulitiByte commented 2 months ago

Hi. Please download all splits of Trainval in Full dataset (v1.0), including the sweeps set. image To replicate our performance, we recommend also downloading OpenDV-YouTube dataset from DriveAGI if possible. You can start with a few YouTubers like J Utah. Feel free to let us know if you encounter any troubles.

Hi, I also want to experiences the pretrained model to see the generated videos based on NuScenes. Howerver I enconter a problem seemingly from dataset.

Loading model from ckpts/vista.safetensors Global seed set to 23 Traceback (most recent call last): File "/data/workspace/code/Vista-main/sample.py", line 254, in img = load_img(each_path, opt.height, opt.width) File "/data/workspace/code/Vista-main/sample.py", line 197, in load_img image = Image.open(file_name) File "/root/anaconda3/envs/vista/lib/python3.9/site-packages/PIL/Image.py", line 3277, in open fp = builtins.open(filename, "rb") FileNotFoundError: [Errno 2] No such file or directory: '/data/workspace/data/nuScenes/samples/CAM_FRONT/n008-2018-08-01-15-16-36-0400__CAM_FRONT__1533151514512404.jpg'

I only download the first two of all parts. Could you please provide a more detailed dataset organization structure after that I have downloaded all of the 10 parts of Nuscenes. (I note that all part files have directory'sample','sweep', should I manually merge them into the same directory?)

Little-Podi commented 2 months ago

Hi @MulitiByte, thanks for your interest. Yes, I think you need to merge them all to fit the annotations.

MulitiByte commented 2 months ago

Hi @MulitiByte, thanks for your interest. Yes, I think you need to merge them all to fit the annotations.

Thank you so much for your kind an patient reply. I have successfully tested on my own machine. And I have some more questions, I set the configurations for sampling as follows:

action='free', n_rounds=1, n_frames=16, n_conds=3, seed=46, height=320, width=576, cfg_scale=2.5, cond_aug=0.0, n_steps=50, rand_gen=True, low_vram=False

Some produced videos are a little unnatural, could you please provide some advice acorrding to your practical experiences to help to improve the quality? Thanks a lot ! (BTW: dose the parameter n_conds means using the first three frames as the condition for the whole 16 frames under this config?)

Little-Podi commented 2 months ago

Hi @MulitiByte,

could you please provide some advice acorrding to your practical experiences to help to improve the quality?

I think the main factor is the number of frames you use. It is not recommended to set n_frames less than 25 if you have enough GPU memory.

dose the parameter n_conds means using the first three frames as the condition for the whole 16 frames under this config?

Yes, you are right!

Little-Podi commented 2 months ago

I'm closing this issue due to its inactivity. Free feel to reopen it if you have further questions.