search

cong-yi / DMM

An Implicit Parametric Morphable Dental Model
51 stars 6 forks source link

I loaded only 8 scenes (batch) to test the script `train_dmm.py`, but still got `CUDA out of memory` on 3 GTX 3090 #3

Closed poirot0805 closed 1 year ago

poirot0805 commented 1 year ago 
Traceback (most recent call last):
  File "/home/xxl/DMM/train_dmm.py", line 410, in <module>
    main_function(args.experiment_directory, args.continue_from)
  File "/home/xxl/DMM/train_dmm.py", line 336, in main_function
    train_loss, losses_log = dmm_net(latent_vecs, sdf_data, is_on_surf, normal, centers_tensor, indices)
  File "/home/xxl/anaconda3/envs/dmm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxl/DMM/networks/dmm_net.py", line 151, in forward
    losses = component_sdf_normal_loss(component_sdf, coords, is_on_surf, normal,
  File "/home/xxl/DMM/networks/loss.py", line 9, in component_sdf_normal_loss
    gradient = compute_gradient(pred_sdf, xyz)
  File "/home/xxl/DMM/utils/math.py", line 16, in compute_gradient
    grad = torch.autograd.grad(y, [x], grad_outputs=grad_outputs, create_graph=True)[0]
  File "/home/xxl/anaconda3/envs/dmm/lib/python3.9/site-packages/torch/autograd/__init__.py", line 276, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 23.70 GiB total capacity; 22.67 GiB already allocated; 6.88 MiB free; 22.70 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

my pytorch version is torch.1.12.0+cu116