Anyway to run test it on Nvidia GPU 10 GB

[ammaryasirnaich]

Hi there, please can you help me if possible, i want to test it on GPU having 10GB VRAM is it possible?

[ammaryasirnaich]

I use to get an Error while running the eval.sh script! runtime error: CUDA out of memory. Tried to allocate 3.05 GiB (GPU 0; 9.78 GiB total capacity; 5.01 GiB already allocated; 2.51 GiB free; 5.54 GiB reserved in total by PyTorch)

[JiayuYANG]

Hi @ammaryasirnaich

It should be able to run on 10GB GPU. The maximum vram required is around 9GB.

Is there any other process that are using your GPU memory? You can use nvidia-smi to check that.

Cheers, Jiayu.

[ammaryasirnaich]

Hi @JiayuYANG, It think there is nothing else that is running on the GPU. I am running the code in a docker-machine

The below is the output from the nvidia-smi command [

And while I run the eval.sh, I get the output as below

[JiayuYANG]

Hi @ammaryasirnaich

Indeed. Seems the code require more than 10 GB to run. However the maximum memory usage reported by torch.cuda.max_memory_allocated() is less than 9 GB, which is weird. I'll check this problem.

For now, you can reduce the number of views by reducing nsrc=4 in eval.sh. (Which will make performance drop a little bit) Or you can try to delete some intermediate tensors and free up memory on GPU using del tensor1 and torch.cuda.empty_cache(). Or, probably the easiest way, run it on a 11GB GPU:)

Jiayu.

[ammaryasirnaich]

Hi Jiayu, Thanks for the reply, i will look into it as you have mentioned.

Cheers

[lauraset]

Hi jiayu, is it possible to run the CVPNet with a 11 GB GPU? I tried it, but failed.

[JiayuYANG]

Hi @lauraset

Yes, it should be able to run on 11GB GPU.

Jiayu.

[lauraset]

Thanks for your reply. Yes, I found that it's possible to run it with the provided DTU images (160 x 128), but it seems that the network can't work with images (384 x 768) even with a batch size of 1. I note that there are high resolution DTU images and depth images, but I am not clear whether the network (CVPNet) trained with high resolution DTU images (for example, 640 x 512) will outperform that trained with low resolution images (160 x 128). Have you ever tried it?

[JiayuYANG]

Hi @lauraset

Training on higher resolution can provide slightly better performance and better generalization ability. However it consumes more GPU memory. Training the network on 640x512 requires at least 16GB GPUs.

Jiayu.

[lauraset]

Thanks for your help, Jiayu. I notice that you adopted the image pyramid method to reduce the memory requirement in the CVPNet. But this way still uses the fixed cost volume (e.g., D =192), compared to the adjustable one used in the cascade mvsnet. Is it the limitation to apply the CVPNet to high resolution images?

[JiayuYANG]

Hi @lauraset

I suggest you to read our paper more carefully. One of our main contribution in CVP-MVSNet is the use of cost volume pyramid instead of fixed cost volume. It enables us to inference high resolution depth map with less memory consumption. It is similar to the concurrent work cascade mvsnet.

Jiayu.