search

Leon-Sander / Local-Multimodal-AI-Chat

GNU General Public License v3.0
136 stars 83 forks source link

Problem in response with CUDA #11

Closed lauraramoningelan closed 8 months ago

lauraramoningelan commented 8 months ago

Hi! I have changed the code of the youtube video to the new version published here in github and I am having this problem with CUDA. I have a GPU Intel Iris Xe Graphics. With the code in the video it works correctly and, although it takes 10 minutes to respond, it ends up responding correctly. Now, at the beginning it shows me well the chatbot doing streamlit run app.py, and as soon as I write something to make it respond, I get this error and in the browser it appears as if streamlit has been disconnected. Any advice? I can't get it to respond well anymore and I don't know what other solutions to try. I post screenshots of the error below.

Thank you in advance!

code issue code issue2 code issue3

Leon-Sander commented 8 months ago

Intel GPUs do not support CUDA, so you have to run it on CPU it seems. Also 10 minutes would on even be too much on the CPU. It seems to be a problem on windows. For me it works like a charm on linux, but on windows I also face some challenges which I could not fully solve yet.

lauraramoningelan commented 8 months ago

I thought I was running it on CPU but I get the error. I will continue to investigate how to improve the code to make it all work. Thanks for you reply!

Leon-Sander commented 8 months ago

With the parameter "gpu_layers" in the config you set how many layers of the model should offload on the gpu (mistral has 32 layer). In your case you set it to 1, so it tries to offload 1 layer on the GPU. Set it to 0 and it should be fine.

lauraramoningelan commented 8 months ago

Thank you very much! You were right, I no longer get the CUDA error but I have a new one :'D; programming stuff hahaha. Another question I have is: can I see everything we are storing in the database? Chats and pdfs? So I can upload my pdfs and the user can chat only with those pdfs.

Leon-Sander commented 8 months ago

If you would deploy the code as it is, then everyone with access to it, would use the same database, and see all the chats and be able to use all ingested pdf files. There is no user management build in right now.