RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
Apache License 2.0
12.32k
stars
838
forks
source link
train.py: "ninja build stopped" (lcudart, cicc, etc. not found) #104
Is the 4090 supported, or do I need to be using a different GPU altogether?
If the docker container I used is not the right one, which one should I use instead?
If train.py doesn't find the location of the CUDA files it needs, how do I manually specify the correct paths? How do I find out which path is the correct one? What file do I search for? If several paths are being found, how do I know which one to pick? And how do I tell train.py which one to use? Is LD_LIBRARY_PATH the right env var to set? Should I set it using os.environ right inside train.py?
I'm having trouble running the train.py file.
I keep getting erros when building the cuda extensions:
ninja: build stopped: subcommand failed
Errors like this:
/usr/bin/ld: cannot find -lcudart
sh: 1: cicc: not found
It looks like torch doesn't find the cuda library files.
So I tried manually searching for those files, and then setting the env LD_LIBRARY_PATH, like this:
With some tweaking of env variables and creating symlinks, I was able to get past one error - but only to encounter the next one.
The server I used is using a NVIDIA GeForce RTX 4090.
I used this image:
nvidia/cuda:11.4.3-devel-ubuntu20.04
I installed these versions:
pip install torch -f https://download.pytorch.org/whl/cu111/torch_stable.html deepspeed==0.7.0 pytorch-lightning==1.9.2
And ran the train file like this:
Questions:
LD_LIBRARY_PATH
the right env var to set? Should I set it using os.environ right inside train.py?Thank you.