Running out of memory of comparatively lower resolution

[iraj465]

Config / Flags:
---------
iter 500
batch 1
spp 1
layers 1
train_res [512, 512]
display_res [512, 512]
texture_res [1024, 1024]
display_interval 0
save_interval 100
learning_rate [0.03, 0.005]
custom_mip False
background white
loss logl1
out_dir out/manual_shoe-9-1024
config configs/nerd_shoe-9.json
ref_mesh /opt/users/saptarshi.majumder/tmp/camera-ds/colmap-1024/shoe-9
base_mesh None
validate True
n_samples 12
bsdf pbr
denoiser bilateral
denoiser_demodulate True
mtl_override None
dmtet_grid 128
mesh_scale 2.5
envlight None
env_scale 1.0
probe_res 256
learn_lighting True
display [{'bsdf': 'kd'}, {'bsdf': 'ks'}, {'bsdf': 'normal'}]
transparency False
lock_light False
lock_pos False
sdf_regularizer 0.2
laplace relative
laplace_scale 3000.0
pre_load True
no_perturbed_nrm False
decorrelated False
kd_min [0.03, 0.03, 0.03]
kd_max [0.8, 0.8, 0.8]
ks_min [0, 0.08, 0.0]
ks_max [0, 1.0, 1.0]
nrm_min [-1.0, -1.0, 0.0]
nrm_max [1.0, 1.0, 1.0]
clip_max_norm 0.0
cam_near_far [0.1, 1000.0]
lambda_kd 0.1
lambda_ks 0.05
lambda_nrm 0.025
lambda_nrm2 0.25
lambda_chroma 0.025
lambda_diffuse 0.15
lambda_specular 0.0025
random_textures True
---------
DatasetLLFF: 72 images with shape [1024, 1024]
DatasetLLFF: auto-centering at [-0.04074487  2.0440297   1.4126209 ]
DatasetLLFF: 72 images with shape [1024, 1024]
DatasetLLFF: auto-centering at [-0.04074487  2.0440297   1.4126209 ]
/opt/users/saptarshi.majumder/tmp/miniconda3/envs/nvdiffrec/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Cuda path /usr
Segmentation fault

Hi, i'm getting segmentation fault on lower resolution images too, can you please give any insights as to why this is happening?

[jmunkberg]

Hello, What GPU is this on, or more precisely, how much memory does your GPU have?

For memory consumption, you can log the usage using nvidia-smi --query-gpu=memory.used --format=csv -lms 100 when you run to get a feel for the usage. Memory is a function of image resolution, batch size and if you have depth peeling enabled. We ran the results in the paper using GPUs with 32+GB of memory.

We rely on nvdiffrast for differentiable rasterization, and rasterizing full screen buffers, so for very high image resolution, the memory cost is significant in the public code base. Easiest way to reduce memory (at the cost of quality) is to reduce the image resolution and/or lower the batch size.

[iraj465]

I'm using an A100 GPU with 40GB memory, so it should be able to load 1024x1024 images in memory? I even reduced the batch size to 2 and 4 respectively but still got segmentation fault.