Error when run LlamaIndex_Mistral7B-RAG.ipynb in Jupyter Notebook.

[chanwitkepha]

According to README file https://github.com/marklysze/LlamaIndex-RAG-Linux-CUDA/blob/main/README.md

When I install from environment files and activate conda.

conda env create -f environment.yml
conda activate LlamaIndexRAGLinux

I install Jupyter Notebook and run.

pip install pip install  notebook ipykernel
jupyter-notebook --allow-root --no-browser --ip=0.0.0.0 --port=27000 --NotebookApp.token='1a4c347ef...........'

Then I run LlamaIndex_Mistral7B-RAG.ipynb in Jupyter Notebook. I select Kernel from Preferred Session (LlamaIndex_Mistral7B-RAG.ipynb)

When I run Code in Notebook at step 3

import torch

from llama_index.llms import LlamaCPP
from llama_index.llms.llama_utils import messages_to_prompt, completion_to_prompt
llm = LlamaCPP(
    model_url=None, # We'll load locally.
    model_path='./Models/mistral-7b-instruct-v0.1.Q6_K.gguf', # 6-bit model
    temperature=0.1,
    max_new_tokens=1024, # Increasing to support longer responses
    context_window=8192, # Mistral7B has an 8K context-window
    generate_kwargs={},
    # set to at least 1 to use GPU
    model_kwargs={"n_gpu_layers": 33}, # 33 was all that was needed for this and that fit with the RTX 3090's 24GB of memory
    messages_to_prompt=messages_to_prompt,
    completion_to_prompt=completion_to_prompt,
    verbose=True
)

It show error output.

ImportError                               Traceback (most recent call last)
Cell In[3], line 1
----> 1 import torch
      3 from llama_index.llms import LlamaCPP
      4 from llama_index.llms.llama_utils import messages_to_prompt, completion_to_prompt

File ~/anaconda3/envs/LlamaIndexRAGLinux/lib/python3.12/site-packages/torch/__init__.py:235
    233     if USE_GLOBAL_DEPS:
    234         _load_global_deps()
--> 235     from torch._C import *  # noqa: F403
    237 # Appease the type checker; ordinarily this binding is inserted by the
    238 # torch._C module initialization code in C
    239 if TYPE_CHECKING:

ImportError: /home/devteam/anaconda3/envs/LlamaIndexRAGLinux/lib/python3.12/site-packages/torch/lib/../../../../libcusparse.so.12: symbol __nvJitLinkGetLinkedCubin_12_2 version libnvJitLink.so.12 not defined in file libnvJitLink.so.12 with link time reference

Please advise how to solve this error, Thank you.

[chanwitkepha]

Additional information.

LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda-12.1/lib64:/usr/local/cuda-11.8/lib64:

 ls -l /usr/local/

total 56
drwxr-xr-x  3 root root 12288 ม.ค.   5 21:23 bin
lrwxrwxrwx  1 root root    22 มิ.ย.  28  2023 cuda -> /etc/alternatives/cuda
lrwxrwxrwx  1 root root    25 ก.ค.  27  2023 cuda-11 -> /etc/alternatives/cuda-11
drwxr-xr-x 15 root root  4096 ก.ค.  27  2023 cuda-11.8
lrwxrwxrwx  1 root root    25 มิ.ย.  28  2023 cuda-12 -> /etc/alternatives/cuda-12
drwxr-xr-x 15 root root  4096 มิ.ย.  28  2023 cuda-12.1
drwxr-xr-x  2 root root  4096 มี.ค.   1  2018 etc
drwxr-xr-x  2 root root  4096 มี.ค.   1  2018 games
drwxr-xr-x  9 root root  4096 เม.ย. 13  2021 include
drwxr-xr-x 12 root root 12288 ธ.ค.   9  2022 lib
lrwxrwxrwx  1 root root     9 พ.ค.  11  2018 man -> share/man
drwxr-xr-x  2 root root  4096 มี.ค.   1  2018 sbin
drwxr-xr-x 15 root root  4096 ต.ค.  21 19:41 share
drwxr-xr-x  2 root root  4096 มี.ค.   1  2018 src

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0

[chanwitkepha]

I try again by install pytorch stable 2.2 pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 And test again, It's not work

ImportError                               Traceback (most recent call last)
Cell In[6], line 1
----> 1 import torch
      3 from llama_index.llms import LlamaCPP
      4 from llama_index.llms.llama_utils import messages_to_prompt, completion_to_prompt

File ~/anaconda3/envs/LlamaIndexRAGLinux/lib/python3.12/site-packages/torch/__init__.py:27
     24     return sys.modules.get("torch._meta_registrations", None) is object
     26 from ._utils import _import_dotted_name, classproperty
---> 27 from ._utils import _functionalize_sync as _sync
     28 from ._utils_internal import get_file_path, prepare_multiprocessing_environment, \
     29     USE_RTLD_GLOBAL_WITH_LIBTORCH, USE_GLOBAL_DEPS
     31 # TODO(torch_deploy) figure out how to freeze version.py in fbcode build

ImportError: cannot import name '_functionalize_sync' from 'torch._utils' (/home/devteam/anaconda3/envs/LlamaIndexRAGLinux/lib/python3.12/site-packages/torch/_utils.py)

So I went back to current version in file environment.yml

Please help, Thank you.

[marklysze]

Hi @chanwitkepha, would you be able to run conda list so we can see the versions and I can compare with mine?

[chanwitkepha]

Hi @chanwitkepha, would you be able to run conda list so we can see the versions and I can compare with mine?

Here is the list of packages that show with conda list.

LlamaIndexRAGLinux-COnda-List.txt

[marklysze]

Thanks @chanwitkepha, is it possible to run the following in your condo environment to downgrade from python 3.12 to 3.11?

conda install python=3. 11

I have seen an issue that pytorch is about to support 3.12 but not yet. Though I don't know why it is working for me. I will also double check but in the meantime if you can try 3.11 please.

[chanwitkepha]

Ok I will try it again.

[chanwitkepha]

Thanks @chanwitkepha, is it possible to run the following in your condo environment to downgrade from python 3.12 to 3.11?

conda install python=3. 11

I have seen an issue that pytorch is about to support 3.12 but not yet. Though I don't know why it is working for me. I will also double check but in the meantime if you can try 3.11 please.

After tryingconda install python=3.11 It have many conflicts then conda cannot install it.

Could you change your environment.yml file for compatible with python 3.11 so I can try to install conda environment again? Thank you.

[marklysze]

Hi @chanwitkepha, I've just run the Mistral 7B notebook with the environment as it was (Python 3.12.1 and PyTorch 2.1.0) and it ran okay. I am running it in Visual Studio Code and selecting the Conda environment as the kernel.

I was wondering if it is a Nvidia CUDA library error - can you run nvidia-smi and see what version shows? Also, what card are you using?

[chanwitkepha]

This is output from my nvidia-smi and nvcc --version.

[chanwitkepha]

I run on Jupyter Notebook

[linuaries]

My issue is llama-index failure after conda install pytorch. And solved by install anything by pip only.

[marklysze]

Thanks for sharing your experience @linuaries, that could be tried - just a note that the recent LlamaIndex libraries have been split into legacy and core so the code won't run as is. I am looking into updating it and will update the repository with a version that works then.

@chanwitkepha, the error does look like a cuda library issue, I'm wondering if there's a difference between cuda v12.1 which you have and version v12.2 which I have? Is it possible to try to update nvidia cuda drivers, as per nvidia-smi to v12.2? The conda list showed version of cuda libraries to be 12.2 so this may be an issue with incompatible libraries (12.2) vs driver (12.1).

[chanwitkepha]

Thanks for sharing your experience @linuaries, that could be tried - just a note that the recent LlamaIndex libraries have been split into legacy and core so the code won't run as is. I am looking into updating it and will update the repository with a version that works then.

@chanwitkepha, the error does look like a cuda library issue, I'm wondering if there's a difference between cuda v12.1 which you have and version v12.2 which I have? Is it possible to try to update nvidia cuda drivers, as per nvidia-smi to v12.2? The conda list showed version of cuda libraries to be 12.2 so this may be an issue with incompatible libraries (12.2) vs driver (12.1).

Thank you, I will upgrade this old machine from Ubuntu 18.04.6 to 22.04 + CUDA 12.2 (or 12.3) and try again.

[marklysze]

Thanks @chanwitkepha, please try 12.2 so it matches the libraries

[marklysze]

Hi @chanwitkepha (and @linuaries) - I'm close to finishing the update to the latest llama-index version (core) and that includes an update to llama-cpp-python as well.

It is advised, by llama-index, that a brand new environment is created and I did this and installed the nvidia cuda toolkit (Base Installer instructions here: https://developer.nvidia.com/cuda-downloads

This is now Cuda v12.3.

I will update the environment.yml file as well.

It is working for me and hopefully will work for you, too. Please wait for the update over the next hour before cloning.

[chanwitkepha]

Hi @chanwitkepha (and @linuaries) - I'm close to finishing the update to the latest llama-index version (core) and that includes an update to llama-cpp-python as well.

It is advised, by llama-index, that a brand new environment is created and I did this and installed the nvidia cuda toolkit (Base Installer instructions here: https://developer.nvidia.com/cuda-downloads

This is now Cuda v12.3.

I will update the environment.yml file as well.

It is working for me and hopefully will work for you, too. Please wait for the update over the next hour before cloning.

Hi, I use git pull and try it again, It seem this environment.yml can work with CUDA 12.1 in my old Ubuntu 18.04.6 LTS.

Now I can pass step 3 but still have error in step 8. It seem my First GPU have too small memory,

But my PC have 2 GPU (GTX 1080) How can I use both GPU for processing this Mistral 7B Models.

[marklysze]

Hi @chanwitkepha, it may be trying to load too many layers onto the GPU. For this n_gpu_layers value, try setting it lower, such as 10 and then if that leaves you a lot of VRAM (check with nvidia-smi) then you can increase it.

Note: The last line visible in my screenshot above says it found 1 CUDA device, I would think you should see this line and it should say 2 CUDA devices. So, I'm not 100% sure if it is using your cards correctly.