araistrick
commented
7 months ago Hello!
3 days is definitely not expected. Are you sure that rendering is using your GPU? Typically I observe ~1hr to generate the scene and 3-5min of GPU rendering per frame, so more like 10hrs latency for a high quality video. If a noticable amount of render noise is acceptable then you can reduce this to 40sec /frame or about 3hr total. We find these costs acceptable for ray-traced cgi given one can run many many videos at once in paralell, going any faster will probably require switching to unreal or another realtime renderer (which we are working on better support for).
With 3-day renders though I would recommend checking that the GPU is enabled and that nvidia-smi shows you are getting good GPU utilization. I will also release a branch ASAP that may increase your GPU utilization if it is currently low, we've done some optimizing.
There are certainly ways to improve speed. First of all, make sure to disable stereo if you dont need stereo video - rendering for two cameras doubles the render cost. Also turn off the ground truth or use blender ground truth if either of these are acceptable for your usecase. Besides these the easiest step is to decrease the number of raytracing samples; you can do this either by increasing render_image.adaptive_threshold = 0.005 to 0.01 or even 0.05 in base.gin depending on how much render noise is acceptable to you. 0.01 is generally fine for outdoor scenes but will look noisy underwater. 0.05 is much faster (as low as 40sec/frame) but noticably noisy in shadows. You can also look in the ConfiguringInfinigen docs to decrese image resolution or mesh resolution if this would be acceptable
The system works best when running one job per GPU, but each job takes ~20GB VRAM (i.e. fits a 3090/4090 or L40 well - this is what we use). Rendering jobs do not usually need the full 32GB or 80GB of a large deep learning GPU. In theory you can split one such GPU into two 40GB GPUs, which is what LocalScheduleHandler.jobs_per_gpu=2 attempts to do, but the current implementation of that feature just assigns two jobs to each gpu, it doesnt limit each to take half the VRAM etc, so the jobs can end up overflowing and both dying.
RE your parameter questions, apologies that there is not yet better docs for these. Here are answers to the ones you mentioned:
- num_scenes affects how many videos will be made. This should generally be set to 100 or more, so that the system slowly works through many videos. However it should not cause crashes, since only
num_concurrentscenes will run at once. If there are crashes you should adjustlocal_256GB.gin's num_concurrent to a lower value - i recommend setting it to (TOTAL_RAM_GIGABYTES / 20), since each scene needs about 20GB. - using local_256GB runs more scenes at once so it may (?) decrease the performance per scene due to CPU contention. but it should increase overall throughput.
- If you are getting very very significant slowdown for this I have one hypothesis. I believe there might possibly be a bug in the local_256GB version wherein each of N blender instance tries to run multithreaded code on all M available CPUs, so when there are multiple blenders running there is some overhead from too many processes (N M rather than N (TOTAL // M)). but im not sure how severe this would be. In our own rendering we almost always use the
slurm.ginvariant on a cluster which restricts which CPUs are visible to which processes. When using external blender rather than pip blender there is a workaround for this where you add--threads 4or something similar to theget_cmdofjob_funcs.pyand it will tell blender to spawn less threads and share more efficiently, but you have to switch to theBlender Python Scriptversion of installation for this to work.- RE jobs_per_gpu I have never personally gotten it to run efficiently with any setting other than 1 job per GPU. If at all possible I recommend sticking to this, or if you are using AWS I recommend getting 24GB or 32GB GPUs as these wont be overly wasted. I have not yet developed a good config for running efficently on 80GB GPUs, they are just somewhat overkill for blender afaik. Although one could potentially split them up into 4 20GB GPUs using https://www.nvidia.com/en-us/technologies/multi-instance-gpu/.
- If you are getting very very significant slowdown for this I have one hypothesis. I believe there might possibly be a bug in the local_256GB version wherein each of N blender instance tries to run multithreaded code on all M available CPUs, so when there are multiple blenders running there is some overhead from too many processes (N M rather than N (TOTAL // M)). but im not sure how severe this would be. In our own rendering we almost always use the
Sorry for the delay in replying, and for the issues that have come up here. Im very keen to help get it running smoothly for you. Im going to release some optimizations ive been working on as a beta branch shortly and I will followup here when that happens. In may-june I also plan to do a great deal more optimization and simplification of the main rendering system.
bbbbubble
mazeyu
Describe the bug
Hi, @araistrick Thanks for this excellent work! I want to inquire about the speed of video generation with InfiniGen. Under default settings (24fps, 192 frames), which means generating an 8-second video, even with GPU enabled for cuda_terrian and rendering, my machine took 3 days. Because I plan to generate a large number of videos with InfiniGen, this speed is unacceptable. Therefore, may i ask:
num_scenesto a value other than1seems easily causes crashes? （2）When running on a single machine with a single GPU card，compared tolocal_64GB, setting it tolocal_256GBslows down the speed of individual tasks? （3）I have no idea how to set warmup_sec. （4）If LocalScheduleHandler.jobs_per_gpu are set according toVRAM/20GB, it is also easy to encounterGPU out of memorycrashes, is that due to shortrender failures, which then trigger backuprender?Reproduce
This is command I use:
python -m infinigen.datagen.manage_jobs --num_scenes 200 --pipeline_configs monocular_video cuda_terrain local_256GB --warmup_sec 6000 --configs video high_quality_terrain --output_folder outputs/my_videosPlatform