search

NVlabs / Deep_Object_Pose

Deep Object Pose Estimation (DOPE) – ROS inference (CoRL 2018)
Other
1.03k stars 287 forks source link

Loss goes to Nan #268

Open prrw opened 2 years ago

prrw commented 2 years ago

Loss goes to Nan from 4th Step of 1st Epoch. But the same training script works well on another computer with another graphic card. Requirement list is same for both. Both are Windows.

Works here: NVIDIA Quadro M2200 Driver version: 471.11 CUDA Version: 11.4

Does not work here as loss goes to nan: NVIDIA RTX A4000 Driver version: 512.59 CUDA Version: 11.6 (CUDA 11.2 is also installed but 11.6 is active)

Also before the loss goes to Nan, it takes lot of time like 30 mins to start the training which is not the case in another PC.

Requirements:

blas                      1.0                         mkl
certifi                   2022.9.14        py37haa95532_0
cudatoolkit               10.2.89              h74a9793_1
freetype                  2.10.4               hd328e21_0
intel-openmp              2021.4.0          haa95532_3556
jpeg                      9e                   h2bbff1b_0
lerc                      3.0                  hd77b12b_0
libdeflate                1.8                  h2bbff1b_5
libpng                    1.6.37               h2a8f88b_0
libtiff                   4.4.0                h8a3f274_0
libwebp                   1.2.2                h2bbff1b_0
lz4-c                     1.9.3                h2bbff1b_1
mkl                       2021.4.0           haa95532_640
mkl-service               2.4.0            py37h2bbff1b_0
mkl_fft                   1.3.1            py37h277e83a_0
mkl_random                1.2.2            py37hf11a4ad_0
ninja                     1.10.2               haa95532_5
ninja-base                1.10.2               h6d14046_5
numpy                     1.21.5           py37h7a0a035_3
numpy-base                1.21.5           py37hca35cd5_3
opencv-python             4.6.0.66                 pypi_0    pypi
pillow                    9.2.0            py37hdc2b20a_1
pip                       22.1.2           py37haa95532_0
protobuf                  3.20.1                   pypi_0    pypi
pyconfigparser            1.0.5                    pypi_0    pypi
python                    3.7.0                hea74fb7_0
pytorch                   1.6.0           py3.7_cuda102_cudnn7_0    pytorch
pyyaml                    6.0                      pypi_0    pypi
setuptools                63.4.1           py37haa95532_0
six                       1.16.0             pyhd3eb1b0_1
tensorboardx              2.5.1                    pypi_0    pypi
tk                        8.6.12               h2bbff1b_0
torchvision               0.7.0                py37_cu102    pytorch
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
wheel                     0.37.1             pyhd3eb1b0_0
wincertstore              0.2              py37haa95532_2
xz                        5.2.5                h8cc25b3_1
zlib                      1.2.12               h8cc25b3_3
zstd                      1.5.2                h19a0ad4_0

I feel is the NVIDIA CUDA version issue. What can go wrong ?