CUDA OOM on A100 with SV3D

[jclarkk]

Tried allocating 67371012.07 GiB which seemed a bit excessive :)

Traceback (most recent call last): File "Dev/GRM/test.py", line 585, in main(args) File "Dev/GRM/test.py", line 508, in main sv3d_gs( File "Dev/GRM/test.py", line 472, in sv3d_gs images2gaussian(images, c2ws, fxfycxcy, grm_model, f'./{cache_dir}/{name}_gs.ply', f'{cache_dir}/{name}.mp4', f'{cache_dir}/{name}_mesh.ply', fuse_mesh=fuse_mesh) File "Dev/GRM/test.py", line 242, in images2gaussian gs_rendering = model.gs_renderer.render(latent=gs, File ".local/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 121, in decorate_fwd return fwd(*_cast(args, cast_inputs), _cast(kwargs, cast_inputs)) File "Dev/GRM/model/render/gaussian_renderer.py", line 40, in render renderings, depths, alphas = deferred_bp(xyz, features, scaling, rotation, File "Dev/GRM/model/render/deferred_bp.py", line 163, in deferred_bp return DeferredBP.apply( File ".local/lib/python3.10/site-packages/torch/autograd/function.py", line 553, in apply return super().apply(*args, *kwargs) # type: ignore[misc] File "Dev/GRM/model/render/deferred_bp.py", line 72, in forward render_results = render(pc, patch_size, patch_size, C2W[i, j], new_fxfycxcy) File "Dev/GRM/model/render/gaussian_utils.py", line 653, in render rendered_image, radii, rendered_depth, rendered_alpha = rasterizer( File ".local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) File ".local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, *kwargs) File ".local/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 210, in forward return rasterize_gaussians( File ".local/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 32, in rasterize_gaussians return _RasterizeGaussians.apply( File ".local/lib/python3.10/site-packages/torch/autograd/function.py", line 553, in apply return super().apply(args, *kwargs) # type: ignore[misc] File ".local/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 92, in forward num_rendered, color, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(args) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 67371012.07 GiB. GPU 0 has a total capacity of 39.39 GiB of which 34.88 GiB is free. Including non-PyTorch memory, this process has 4.50 GiB memory in use. Of the allocated memory 3.68 GiB is allocated by PyTorch, and 269.31 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[jclarkk]

My bad, noticed now that I installed Inria's diff-gaussian-rasterization instead of the one in the repo.

[thucz]

@jclarkk Sorry to bother you! It seems that you have the GRM code. GRM code is now withdrawed. Can you share a copy with me? My email is zhengchenecho@gmail.com.