To avoid the Lack of GPU Memory

[Realitian]

Hi

When I run the bash render_to_screen.sh, I'v got following error. GPUassert: too many resources requested for launch network_eval.cu 292 Looks like my GPU memory is not enough, but is there any config to reduce the GPU memory size?

-- full log is ---

auto log path: logs/paper/finetune/Synthetic_NeRF_Lego
{'checkpoint_interval': 50000, 'chunk_size': 4000, 'distilled_cfg_path': 'cfgs/paper/distill/Synthetic_NeRF_Lego.yaml', 'distilled_checkpoint_path': 'logs/paper/distill/Synthetic_NeRF_Lego/checkpoint.pth', 'initial_learning_rate': 0.001, 'iterations': 1000000, 'l2_regularization_lambda': 1e-06, 'learing_rate_decay_rate': 500, 'no_batching': True, 'num_rays_per_batch': 8192, 'num_samples_per_ray': 384, 'occupancy_cfg_path': 'cfgs/paper/pretrain_occupancy/Synthetic_NeRF_Lego.yaml', 'occupancy_log_path': 'logs/paper/pretrain_occupancy/Synthetic_NeRF_Lego/occupancy.pth', 'perturb': 1.0, 'precrop_fraction': 0.5, 'precrop_iterations': 0, 'raw_noise_std': 0.0, 'render_only': False, 'no_color_sigmoid': False, 'render_test': True, 'render_factor': 0, 'testskip': 8, 'deepvoxels_shape': 'greek', 'blender_white_background': True, 'blender_half_res': False, 'llff_factor': 8, 'llff_no_ndc': False, 'llff_lindisp': False, 'llff_spherify': False, 'llff_hold': False, 'print_interval': 100, 'render_testset_interval': 10000, 'render_video_interval': 100000000, 'network_chunk_size': 65536, 'rng_seed': 0, 'use_same_initialization_for_all_networks': False, 'use_initialization_fix': False, 'num_importance_samples_per_ray': 0, 'model_type': 'multi_network', 'random_direction_probability': -1, 'von_mises_kappa': -1, 'view_dependent_dropout_probability': -1}
Using GPU: NVIDIA GeForce GTX 1660
/home/kevin/Documents/kilonerf-master/utils.py:254: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return np.array([[float(w) for w in line.strip().split()] for line in open(path)]).astype(np.float32)
Loaded a NSVF-style dataset (138, 800, 800, 4) (138, 4, 4) (0,) data/nsvf/Synthetic_NeRF/Lego
(100,) (13,) (25,)
Converting alpha to white.
global_domain_min: [-0.67 -1.2  -0.37], global_domain_max: [0.67 1.2  1.03], near: 2.0, far: 6.0, background_color: tensor([1., 1., 1.])
Loading logs/paper/finetune/Synthetic_NeRF_Lego/checkpoint_1000000.pth
Loading occupancy grid from logs/paper/pretrain_occupancy/Synthetic_NeRF_Lego/occupancy.pth
GPUassert: too many resources requested for launch network_eval.cu 292

Thanks

[creiser]

Hi,

this is probably again due to an insufficient number of registers as it was the case in #1. Can you try to decrease the number of threads by e.g. adding network_eval_num_threads = 512 after https://github.com/creiser/kilonerf/blob/e79af85198f579d7b588d4ba9aae598384859dd8/fast_kilonerf_renderer.py#L150 (use even lower values if it still does not work)

Have you pulled the most recent commit (problem is already fixed there for RTX 2080 Ti). Which GPU you are using?

[Realitian]

Using GPU: NVIDIA GeForce GTX 1660, I got face this.

[kaihodge]

Ran across this issue as well on GeForce RTX 3080. network_eval_num_threads = 512 fixed.

The device compute_capability is (8, 6) for my 3080.

referring to https://en.wikipedia.org/wiki/CUDA#Version_features_and_specifications a few values have gone down for 8.6.

Once set to 512, runs smoothly (11ms).