search

mimbres / neural-audio-fp

https://mimbres.github.io/neural-audio-fp
MIT License
179 stars 25 forks source link

Getting training loss value as nan and val loss as nan #33

Closed kasireddygariDineshKumarReddy closed 2 years ago

kasireddygariDineshKumarReddy commented 2 years ago

I'm using NVIDIA RTX A5000 . I'm getting loss values as nan during training with default values.I'm getting this ..please clarify and solve the issue W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation. Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal : Value 'sm_86' is not defined for option 'gpu-name'

mimbres commented 2 years ago

@kasireddygariDineshKumarReddy It seems FAISS requires to be recompiled with the 'sm_86' tag. At the time I was compiling it, the gpu type was not taken into account. I recommend you to try conda install faiss-gpu or faiss-cpu(search speed is slow) first. If conda install faiss-gpu doesn't work, you will have to build from source as described in: https://github.com/facebookresearch/faiss/blob/main/INSTALL.md.

kasireddygariDineshKumarReddy commented 2 years ago

I had ran conda install faiss-gpu successfully but cant get rid of nan loss

mimbres commented 2 years ago

@kasireddygariDineshKumarReddy Sorry, forget my answer. It's not a faiss related issue. The same issue as yours was previously solved in https://github.com/mimbres/neural-audio-fp/issues/20.

kasireddygariDineshKumarReddy commented 2 years ago

I have ran $docker pull mimbres/neural-audio-fp:cuda11.2.0-cudnn8 this command still problem is same. I'm still getting W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation. Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal : Value 'sm_86' is not defined for option 'gpu-name' and Loss as NAN

mimbres commented 2 years ago

Your issue is related to specific GPU type, CUDA toolkit version and Driver version. So I cannot help you much for the basic setup. https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html https://docs.nvidia.com/deploy/cuda-compatibility/index.html According to the compatiability table, A5000 is Ampere architecture GPU (sm_80, sm_86, sm_87). Minimum Driver version is '450.36.06', and CUDA vesrion > 11.0. 11.0 is buggy, so 11.2 would be a good choice. 11.2 is also the most preferred version for tensorflow > 2.5. I also recommend you to check first if any other tensorflow code runs correctly on your environment. If it works, you can manually install the requirements (of this repo) step by step.

mimbres commented 2 years ago

https://github.com/tensorflow/tensorflow/issues/45044 reinstall tensorflow 2.6 may help...

kasireddygariDineshKumarReddy commented 2 years ago

please let me clarify what are the updates to do in yaml file to avoid incompatibilities.

mimbres commented 2 years ago

Your frustration is understandable. However, I can't help you this way. First of all, can you check the results of the methods I recommended earlier? Are the other TensorFlow examples working well, the correct driver version is installed, etc. If that's clear, the training part of the repo can be run in any environment without this docker. Keep in mind that I'm limited in helping out with issues that I can't reproduce.

P.S. I also recommend you to try an official Tensorflow Docker image, and follow the instruction in this repo for manual installation.