failed: CUDA_ERROR_OUT_OF_MEMORY

[Double-zh]

ns-export-semantics semantic-pointcloud --load-config /home/catas/FruitNeRF/output3/output2/output1/fruit_nerf/2024-08-27_173038/config.yml --output-dir /home/catas/FruitNeRF/output3/output2/output1/fruit_nerf/2024-08-27_173038/nerfstudio_models/step-000029999.ckpt --use-bounding-box True

Hello, I encountered an error message "failed: CUDA-ERROROD_OUT_SEMORY" while running this command. How can I resolve this issue. My device is NVIDIA GeForce RTX 4060 8G, can it run?

(fruit) catas@catas:~/FruitNeRF/segmentation$ ns-export-semantics semantic-pointcloud --load-config /home/catas/FruitNeRF/output3/output2/output1/fruit_nerf/2024-08-27_173038/config.yml --output-dir /home/catas/FruitNeRF/output3/output2/output1/fruit_nerf/2024-08-27_173038/nerfstudio_models/step-000029999.ckpt --use-bounding-box True /home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torchmetrics/utilities/prints.py:62: FutureWarning: Importing PeakSignalNoiseRatio from torchmetrics was deprecated and will be removed in 2.0. Import PeakSignalNoiseRatio from torchmetrics.image instead. _future_warning( Loading latest checkpoint from load_dir ✅ Done loading checkpoint from /home/catas/FruitNeRF/output3/output2/output1/fruit_nerf/2024-08-27_173038/nerfstudio_models/step-000029999.ckpt /home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.) return _VF.meshgrid(tensors, kwargs) # type: ignore[attr-defined] ☁ Computing Point Cloud ☁ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0% --:-- Traceback (most recent call last): File "/home/catas/anaconda3/envs/fruit/bin/ns-export-semantics", line 8, in sys.exit(entrypoint()) File "/home/catas/FruitNeRF/fruit_nerf/scripts/exporter.py", line 135, in entrypoint tyro.cli(Commands).main() File "/home/catas/FruitNeRF/fruit_nerf/scripts/exporter.py", line 102, in main pcds = sample_volume( File "/home/catas/FruitNeRF/fruit_nerf/export/exporter_utils.py", line 98, in sample_volume outputs = pipeline.model(ray_bundle) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/catas/FruitNeRF/fruit_nerf/fruit_nerf.py", line 313, in forward fruit_nerf_output = self.get_inference_outputs(ray_bundle, True) File "/home/catas/FruitNeRF/fruit_nerf/fruit_nerf.py", line 228, in get_inference_outputs field_outputs = self.field.forward(ray_samples, render_rgb=render_rgb) File "/home/catas/FruitNeRF/fruit_nerf/fruit_field.py", line 291, in forward density, density_embedding = self.get_density(ray_samples) File "/home/catas/FruitNeRF/fruit_nerf/fruit_field.py", line 184, in get_density h = self.mlp_base(positions_flat).view(ray_samples.frustums.shape, -1) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/container.py", line 215, in forward input = module(input) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/nerfstudio/field_components/encodings.py", line 377, in forward return self.tcnn_encoding(in_tensor) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/tinycudann/modules.py", line 190, in forward output = _module_function.apply( File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/autograd/function.py", line 539, in apply return super().apply(*args, **kwargs) # type: ignore[misc] File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/tinycudann/modules.py", line 102, in forward native_ctx, output = native_tcnn_module.fwd(input, params) RuntimeError: /tmp/pip-req-build-amyotzef/include/tiny-cuda-nn/gpu_memory.h:584 cuMemCreate(&m_handles.back(), n_bytes_to_allocate, &prop, 0)

failed: CUDA_ERROR_OUT_OF_MEMORY

[meyerls]

You can decrease the argument --num_rays_per_batch to 1024 or 512. It does not affect the quality but rather the sampling speed of the volume.

[Double-zh]

Thank you very much. The program is running normally.

[Double-zh]

You can decrease the argument --num_rays_per_batch to 1024 or 512. It does not affect the quality but rather the sampling speed of the volume.

I am very sorry to bother you again. I encountered the same error when I ran the following command. What parameters can I set to reduce the program's demand for video memory?

(fruit) catas@catas:~/FruitNeRF/segmentation$ ns-process-fruit-data --data /home/catas/FruitNeRF/FruitNeRF_Dataset/tree_01/images/ --output-dir /home/catas/FruitNeRF/output --segmentation-class apple

['/home/catas/anaconda3/envs/fruit/bin', '/home/catas/realsense_catkin_ws/devel/lib/python2.7/dist-packages', '/home/catas/catkin_ws/devel/lib/python2.7/dist-packages', '/opt/ros/melodic/lib/python2.7/dist-packages', '/home/catas/anaconda3/envs/fruit/lib/python38.zip', '/home/catas/anaconda3/envs/fruit/lib/python3.8', '/home/catas/anaconda3/envs/fruit/lib/python3.8/lib-dynload', '/home/catas/.local/lib/python3.8/site-packages', '/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages', 'editable.fruit_nerf-0.0.1.finder.__path_hook__', '/home/catas/FruitNeRF/segmentation/grounded_sam/segment_anything'] /home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] final text_encoder_type: bert-base-uncased /home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884 warnings.warn(

[22:39:27]  Done copying images. process_data_utils.py:239 [22:44:21]  Done downscaling images. process_data_utils.py:362 /home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/transformers/modeling_utils.py:1126: FutureWarning: The `device` argument is deprecated and will be removed in v5 of Transformers. warnings.warn( /home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants. warnings.warn( /home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/utils/checkpoint.py:61: UserWarning: None of the inputs have requires_grad=True. Gradients will be None warnings.warn( Before NMS: 23 boxes After NMS: 23 boxes Traceback (most recent call last): File "/home/catas/anaconda3/envs/fruit/bin/ns-process-fruit-data", line 8, in sys.exit(entrypoint()) File "/home/catas/FruitNeRF/fruit_nerf/scripts/process_data.py", line 39, in entrypoint tyro.cli(Commands).main() File "/home/catas/FruitNeRF/fruit_nerf/fruit_nerf_dataset.py", line 319, in main mask = segmentor.run(image_path=image_path, File "/home/catas/FruitNeRF/fruit_nerf/fruit_nerf_dataset.py", line 225, in run return self.model.run(image_path, File "/home/catas/FruitNeRF/fruit_nerf/fruit_nerf_dataset.py", line 170, in run detections.mask = segment( File "/home/catas/FruitNeRF/fruit_nerf/fruit_nerf_dataset.py", line 158, in segment sam_predictor.set_image(image) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/segment_anything_hq/predictor.py", line 62, in set_image self.set_torch_image(input_image_torch, image.shape[:2]) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/segment_anything_hq/predictor.py", line 91, in set_torch_image self.features, self.interm_features = self.model.image_encoder(input_image) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/segment_anything_hq/modeling/image_encoder.py", line 113, in forward x = blk(x) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/segment_anything_hq/modeling/image_encoder.py", line 177, in forward x = self.attn(x) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/segment_anything_hq/modeling/image_encoder.py", line 237, in forward attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (H, W), (H, W)) File "/home/catas/anaconda3/envs/fruit/lib/python3.8/site-packages/segment_anything_hq/modeling/image_encoder.py", line 361, in add_decomposed_rel_pos attn.view(B, q_h, q_w, k_h, k_w) + rel_h[:, :, :, :, None] + rel_w[:, :, :, None, :] **torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU 0 has a total capacty of 7.75 GiB of which 16.06 MiB is free. Process 6035 has 2.23 GiB memory in use. Including non-PyTorch memory, this process has 5.49 GiB memory in use. Of the allocated memory 5.26 GiB is allocated by PyTorch, and 114.80 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF**

[meyerls]

Yes, SAM-HQ does take a lot of VRAM. Either you take a smaller SAM-HQ architecture/model or switch back to the default SAM model from the Grounded SAM instruction site. Both should reduce the VRAM load