search

princeton-vl / infinigen

Infinite Photorealistic Worlds using Procedural Generation
https://infinigen.org
BSD 3-Clause "New" or "Revised" License
5.16k stars 431 forks source link

Reproducing weather simulation results #261

Closed zhihao-lin closed 5 days ago

zhihao-lin commented 5 days ago

Describe the bug

Hi @araistrick, thank you for the excellent work, and the demo video looks fantastic! Currently, I am trying to reproduce the weather effects (e.g. rain, snow) that are demonstrated in the video 1:05, 2:07, but unfortunately, the results I generated are not as good as expected. As a result, I have some questions regarding this issue:

Steps to Reproduce

What command did you run?

I generate the rainy video with the following script:

python -m infinigen.datagen.manage_jobs \
    --output_folder outputs/video_rain --num_scenes 1 --specific_seed 0617 \
    --pipeline_configs local_64GB monocular_video cuda_terrain blender_gt \
    --cleanup big_files --warmup_sec 60000 --config high_quality_terrain \
    --overrides compose_scene.rain_particles_chance=1.0

and the generated frames are provided below: https://github.com/princeton-vl/infinigen/assets/39002023/09fbee13-93dc-4a82-b665-03d07376df83 where the dynamic of rain is not temporally continuous and each frame doesn't look like it's rainy.

Platform

Thanks for your help!

araistrick commented 5 days ago

Hello! The sole issue here (I believe) is the lack of motion blur. We turn this off by default for efficiency and because the ground truth always reflects the non blurred version. You can re-enable it adding 'configure_blender.motion_blur=True' to your --overrides.

Thanks for the question and providing the reference images, lmk if this does not fix.

zhihao-lin commented 5 days ago

Thanks for the prompt response! The generated rainy images look more realistic by adding configure_blender.motion_blur=True after the --overrides, as shown in the first image below. One thing I found interesting is that the ground truth also reflects the motion blur, which could be seen in the depth (second image) and normal (third image).

Image_0_0_0009_0 Depth_0_0_0009_0 SurfaceNormal_0_0_0009_0

zhihao-lin commented 5 days ago

Another thing about the rain simulation is that after applying motion blur, the generation takes much longer time (more than 5.5 hours). I am not sure if this is expected and am wondering if there's any tricks to accelerate.