search

NVIDIA / MinkowskiEngine

Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors
https://nvidia.github.io/MinkowskiEngine
Other
2.43k stars 360 forks source link

cuda is available but GPU isnt used when running training.py #514

Open davodogster opened 1 year ago

davodogster commented 1 year ago

Describe the bug

Hi I successfully setup Mink Docker GPU (on my linux machine with CUDA 11.6) and installed jupyter lab which I port forwarded to my local windows machine. When I run the training.py code it works and I can see the loss decrease but the GPU is not used even though torch.cuda.is_available = True

To Reproduce

docker run --gpus=all -it -p 8045:8888 --shm-size 20G --rm -vpwd:/workspace/MinkowskiEngine/mounted_dir minkowski_engine torch.version, torch.cuda.is_available()

image image

Expected behavior A clear and concise description of what you expected to happen. when I run nvidia-smi in a seperate shell while training I expect it to be using some GPU memory.

Desktop (please complete the following information):

==========System========== Linux-5.14.0-1054-oem-x86_64-with-debian-buster-sid 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0] ==========Pytorch========== 1.12.0 torch.cuda.is_available(): True ==========NVIDIA-SMI========== Driver Version 510.108.03 CUDA Version 11.6 VBIOS Version 94.02.42.00.2F Image Version G001.0000.03.03 GSP Firmware Version N/A ==========NVCC========== ==========CC========== ==========MinkowskiEngine========== 0.5.4 MinkowskiEngine compiled with CUDA Support: True NVCC version MinkowskiEngine is compiled: 11030 CUDART version MinkowskiEngine is compiled: 11030

Additional context Add any other context about the problem here.

guatavita commented 1 year ago

I suggest using cuda 11.3 with torch 1.12.0.

Check this if that helps:

https://github.com/guatavita/FCGF#setting-up-the-requirements