cosmic-snow
commented
1 year ago Maybe this is the same problem as #570?
One comment there says to try: cmake -G "MinGW Makefiles" .. instead of cmake .. before cmake --build . --parallel
Closed gavtography closed 1 year ago
cosmic-snow
commented
1 year ago Maybe this is the same problem as #570?
One comment there says to try: cmake -G "MinGW Makefiles" .. instead of cmake .. before cmake --build . --parallel
gavtography
commented
1 year ago Maybe this is the same problem as #570?
One comment there says to try:
cmake -G "MinGW Makefiles" ..instead ofcmake ..beforecmake --build . --parallel
I did try this already. This is what happens:
$ cmake -G "MinGW Makefiles" ..
CMake Error: Error: generator : MinGW Makefiles
Does not match the generator used previously: Visual Studio 17 2022
Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.Then I tried to rm delete the CMakeCache.txt file, but it says it doesn't exist in the directory, which I don't quite understand. Especially because it shows up in ls.
cosmic-snow
commented
1 year ago Then I'd suggest trying cmake from within a mingw console. That's what I did, anyway.
Open it, then do:
pacman -Syu
pacman -S mingw-w64-x86_64-cmakeAfter that, you might also have to install a compiler if that isn't already present. I've had MSYS2 on my system for a while, I don't remember all the things I installed. 😅
gavtography
commented
1 year ago Then I'd suggest trying cmake from within a mingw console. That's what I did, anyway.
Open it, then do:
pacman -Syu pacman -S mingw-w64-x86_64-cmakeAfter that, you might also have to install a compiler if that isn't already present. I've had MSYS2 on my system for a while, I don't remember all the things I installed. 😅
mingw console as in the cygwin64 terminal I have installed? The command "pacman" isn't found. Is this something I need to install? Can't really find much from a simple quick Google search.
cosmic-snow
commented
1 year ago Sorry, I thought you had MSYS2 installed. It's derived from Cygwin and uses pacman as its package manager.
I haven't used Cygwin in quite a long time, but I think it has its own package manager and maybe there's a cmake package there, too. Can't help you there, though, and no idea if it'll work in the end (although I guess chances are good it will).
Edit: package search on the Cygwin homepage says there is:
gavtography
commented
1 year ago Sorry, I thought you had MSYS2 installed. It's derived from Cygwin and uses
pacmanas its package manager.I haven't used Cygwin in quite a long time, but I think it has its own package manager and maybe there's a
cmakepackage there, too. Can't help you there, though, and no idea if it'll work in the end (although I guess chances are good it will).Edit: package search on the Cygwin homepage says there is:
Well, I'm happy to use MSYS2. Whatever it takes to get this working in python. I installed it and ran:
pacman -Syu
pacman -S mingw-w64-x86_64-cmakeI went back to cygwin64 and ran this:
cmake -G "MinGW Makefiles" ..
then it outputs this:
$ cmake -G "MinGW Makefiles" ..
CMake Error: CMake was unable to find a build program corresponding to "MinGW Makefiles". CMAKE_MAKE_PROGRAM is not set. You probably need to select a different build tool.
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!Maybe I'm lost in the process somewhere. Should I be following the entire README using msys2 instead of cygwin64? or are these separate programs with separate purposes?
cosmic-snow
commented
1 year ago If you're using MSYS2 or Cygwin you probably don't have to specify -G "MinGW Makefiles". I didn't have to when I installed it.
Quick explanation on Cygwin and MSYS2:
And yes, it might be a good idea to just start from scratch to get rid of anything that was produced with MSVC.
gavtography
commented
1 year ago If you're using MSYS2 or Cygwin you probably don't have to specify
-G "MinGW Makefiles". I didn't have to when I installed it.Quick explanation on Cygwin and MSYS2:
- Cygwin makes GNU and Linux stuff work on Windows
MSYS2 is derived from Cygwin and has one key advantage, but that can also make it more confusing. Cygwin builds always depend on a Cygwin runtime. MSYS2 on the other hand can build without that. But it offers several options:
- MSYS2 MSYS console: similar to Cygwin
- MSYS2 MINGW 64-bit console: this is generally what you want to not have that limitation
- there are additional ones for other architectures, e.g. as 32-bit
And yes, it might be a good idea to just start from scratch to get rid of anything that was produced with MSVC.
Alright well I've installed git on MSYS2, followed the README. But now I'm stuck, the command "cmake" isn't found at all anymore. I even tried to install mingw-w64-x86_64-cmake using pacman manually, but still cmake isn't found on MSYS2.
cosmic-snow
commented
1 year ago You need to run the MSYS2 MINGW 64-bit console. As I said, several types of consoles and it can be more confusing. 😉
gavtography
commented
1 year ago You need to run the MSYS2 MINGW 64-bit console. As I said, several types of consoles and it can be more confusing. 😉
Oh boy, right, I see now.
Running cmake .. results in this:
-- Building for: Ninja
-- The CXX compiler identification is unknown
-- The C compiler identification is unknown
CMake Error at CMakeLists.txt:22 (project):
No CMAKE_CXX_COMPILER could be found.
Tell CMake where to find the compiler by setting either the environment
variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
to the compiler, or to the compiler name if it is in the PATH.
CMake Error at CMakeLists.txt:22 (project):
No CMAKE_C_COMPILER could be found.
Tell CMake where to find the compiler by setting either the environment
variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
the compiler, or to the compiler name if it is in the PATH.
-- Configuring incomplete, errors occurred!Then running $ cmake -G "MinGW Makefiles" .. results in this:
CMake Error: Error: generator : MinGW Makefiles
Does not match the generator used previously: Ninja
Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.Also, I appreciate your help and patience 😅
cosmic-snow
commented
1 year ago Run:
cmake ..
cmake --build . --parallelno need for the extra cmake -G "MinGW Makefiles" ..
Aside: oh I see, they (the MSYS2 guys) recommend the UCRT stuff as default now. With what I wrote above, you have to use the MinGW 64-bit console, as the pacman command is for that. If you want to use a different console, you have to install a different package. You can run:
pacman -Ss cmaketo see all options. I'd recommend just sticking with MinGW-64bit for now, though. It's what I did and I made it work that way.
gavtography
commented
1 year ago Run:
cmake .. cmake --build . --parallelno need for the extra
cmake -G "MinGW Makefiles" ..Aside: oh I see, they (the MSYS2 guys) recommend the UCRT stuff as default now. With what I wrote above, you have to use the MinGW 64-bit console, as the pacman command is for that. If you want to use a different console, you have to install a different package. You can run:
pacman -Ss cmaketo see all options. I'd recommend just sticking with MinGW-64bit for now, though. It's what I did and I made it work that way.
Hm, looks like I'm getting a similar error regardless.
$ cmake ..
cmake --build . --parallel
-- The CXX compiler identification is unknown
-- The C compiler identification is unknown
CMake Error at CMakeLists.txt:22 (project):
No CMAKE_CXX_COMPILER could be found.
Tell CMake where to find the compiler by setting either the environment
variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
to the compiler, or to the compiler name if it is in the PATH.
CMake Error at CMakeLists.txt:22 (project):
No CMAKE_C_COMPILER could be found.
Tell CMake where to find the compiler by setting either the environment
variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
the compiler, or to the compiler name if it is in the PATH.
-- Configuring incomplete, errors occurred!
ninja: error: loading 'build.ninja': The system cannot find the file specified.Apologies if I'm missing something obvious or not understanding something completely.
cosmic-snow
commented
1 year ago Now we're at the stage where you need a compiler (mentioned earlier).
pacman -S mingw-w64-x86_64-gccAlso, no need to apologise. Things are complicated.
gavtography
commented
1 year ago Now we're at the stage where you need a compiler (mentioned earlier).
pacman -S mingw-w64-x86_64-gccAlso, no need to apologise. Things are complicated.
Appreciate it. Alright so that seems to have gone successfully. I tried moving onto this:
cd ../../gpt4all-bindings/python
pip3 install -e .pip3 wasn't recognized so I tried this:
pacman -S mingw-w64-x86_64-python
However, pip3 still isn't recognized.
cosmic-snow
commented
1 year ago That's a Linux thing. They had to keep calling the Python 2 version pip and give the Python 3 version a separate name. You typically also see Python invoked as python3 on Linux.
So on Windows it's generally just pip.
gavtography
commented
1 year ago That's a Linux thing. They had to keep calling the Python 2 version
pipand give the Python 3 version a separate name. You typically also see Python invoked aspython3on Linux.So on Windows it's generally just
pip.
Oh sorry, that's my fault, I actually did run the command with regular pip but it's saying pip is not found. Mixed it up in my reply because the README says pip3.
cosmic-snow
commented
1 year ago Hm, is it not in the Python package by default? Looks like it's packaged separately:
pacman -S mingw-w64-x86_64-python-pipAlright, so you might actually have to invoke it as pip3, not sure. But we will see. I went about it slightly differently here.
gavtography
commented
1 year ago Hm, is it not in the Python package by default? Looks like it's packaged separately:
pacman -S mingw-w64-x86_64-python-pip
Ah there we go. Yeah apparently pip is packaged separately for some reason.
That was successful, but unfortunately running the example python script still results in the same error. It opens pyllmodel.py with this:
Exception has occurred: FileNotFoundError
Could not find module 'c:\cygwin64\home\USER\gpt4all\gpt4all-bindings\python\gpt4all\llmodel_DO_NOT_MODIFY\build\libllama.dll' (or one of its dependencies). Try using the full path with constructor syntax.
File "C:\cygwin64\home\USER\gpt4all\gpt4all-bindings\python\gpt4all\pyllmodel.py", line 49, in load_llmodel_library
llama_lib = ctypes.CDLL(llama_dir, mode=ctypes.RTLD_GLOBAL)
File "C:\cygwin64\home\USER\gpt4all\gpt4all-bindings\python\gpt4all\pyllmodel.py", line 55, in <module>
llmodel, llama = load_llmodel_library()
File "C:\cygwin64\home\USER\gpt4all\gpt4all-bindings\python\gpt4all\__init__.py", line 1, in <module>
from .pyllmodel import LLModel # noqa
File "C:\Users\USER\Desktop\bigmantest.py", line 1, in <module>
from gpt4all import GPT4All
FileNotFoundError: Could not find module 'c:\cygwin64\home\USER\gpt4all\gpt4all-bindings\python\gpt4all\llmodel_DO_NOT_MODIFY\build\libllama.dll' (or one of its dependencies). Try using the full path with constructor syntax.pip and pip3 have the same results.
cosmic-snow
commented
1 year ago I went into that over in issue #710. Scroll to: https://github.com/nomic-ai/gpt4all/issues/710#issuecomment-1561781213
The tl;dr is: it might not pick up certain libraries. Simple fix is to put those into Windows' System32 folder. But now that we're here again, I should investigate and come up with a better recommendation.
gavtography
commented
1 year ago I went into that over in issue #710. Scroll to: #710 (comment)
The tl;dr is: it might not pick up certain libraries. Simple fix is to put those into Windows' System32 folder. But now that we're here again, I should investigate and come up with a better recommendation.
Hm, I see. I suppose that is something I could do, but it would probably require a slightly simplified step-by-step explanation on how exactly to do that. It's probably more efficient for you, or anyone else, to investigate and come up with another recommendation as you've suggested, if you'd prefer.
As a matter of fact, it looks like I'm missing even more files in ~/gpt4all/gpt4all-backend/build when I run ls than I did before.
I, and probably many others, would appreciate a little investigation into this, the wait is worth it in the meantime.
cosmic-snow
commented
1 year ago You should have a libllmodel.dll in the build/ directory and a libllama.dll in the build/bin/ directory (that's where they should be, no need to move).
The extra libraries (the dependencies) are in the MSYS2 installation subfolder: msys64\mingw64\bin and called: libstdc++-6.dll, libwinpthread-1.dll and libgcc_s_seh-1.dll. I am investigating a better way, but it'll take some time.
gavtography
commented
1 year ago You should have a
libllmodel.dllin thebuild/directory and alibllama.dllin thebuild/bin/directory (that's where they should be, no need to move).The extra libraries (the dependencies) are in the MSYS2 installation subfolder:
msys64\mingw64\binand called:libstdc++-6.dll,libwinpthread-1.dllandlibgcc_s_seh-1.dll. I am investigating a better way, but it'll take some time.
Just for clarification, I've copy pasted those 3 dll files you mentioned into the System32 folder. Was this supposed to fix it and allow the example script to run? I am still getting the same error in Visual Studio unfortunately.
cosmic-snow
commented
1 year ago Just for clarification, I've copy pasted those 3 dll files you mentioned into the System32 folder. Was this supposed to fix it and allow the example script to run?
Yes, I think it should have.
I am still getting the same error in Visual Studio unfortunately.
Ah, so you're back in Visual Studio now? How are you starting a Python interpreter there?
gavtography
commented
1 year ago Just for clarification, I've copy pasted those 3 dll files you mentioned into the System32 folder. Was this supposed to fix it and allow the example script to run?
Yes, I think it should have.
I am still getting the same error in Visual Studio unfortunately.
Ah, so you're back in Visual Studio now? How are you starting a Python interpreter there?
Right, I run python scripts all the time in Visual Studio. I just run with debugging. Never ever have any issues with Python itself.
from gpt4all import GPT4All
gptj = GPT4All("ggml-gpt4all-j-v1.3-groovy")
messages = [{"role": "user", "content": "Name 3 colors"}]
gptj.chat_completion(messages)
responseI run that code, then it opens pyllmodel.py automatically with this:
Exception has occurred: FileNotFoundError
Could not find module 'c:\cygwin64\home\USER\gpt4all\gpt4all-bindings\python\gpt4all\llmodel_DO_NOT_MODIFY\build\libllama.dll' (or one of its dependencies). Try using the full path with constructor syntax.
File "C:\cygwin64\home\USER\gpt4all\gpt4all-bindings\python\gpt4all\pyllmodel.py", line 49, in load_llmodel_library
llama_lib = ctypes.CDLL(llama_dir, mode=ctypes.RTLD_GLOBAL)
File "C:\cygwin64\home\USER\gpt4all\gpt4all-bindings\python\gpt4all\pyllmodel.py", line 55, in <module>
llmodel, llama = load_llmodel_library()
File "C:\cygwin64\home\USER\gpt4all\gpt4all-bindings\python\gpt4all\__init__.py", line 1, in <module>
from .pyllmodel import LLModel # noqa
File "C:\Users\USER\Desktop\bigmantest.py", line 1, in <module>
from gpt4all import GPT4All
FileNotFoundError: Could not find module 'c:\cygwin64\home\USER\gpt4all\gpt4all-bindings\python\gpt4all\llmodel_DO_NOT_MODIFY\build\libllama.dll' (or one of its dependencies). Try using the full path with constructor syntax.Ah, alright. So that looks to me like in Visual Studio you're using the Cygwin Python interpreter. That would have its own pip. So install that and do the pip install ... again, or switch it to the new MSYS2 Python.
gavtography
commented
1 year ago Ah, alright. So that looks to me like in Visual Studio you're using the Cygwin Python interpreter. That would have its own
pip. So install that and do thepip install ...again, or switch it to the new MSYS2 Python.
Oh, I see what you mean now. I suppose my Python interpreter came from the default Python installer from python.org? So not the Cygwin one. Regardless, I changed the python interpreter on Visual Studio to the MSYS2 one. I wasn't really aware of what exactly a python interpreter was, but now I understand. Anyways, this worked.
For some reason, when I run the script, it spams the terminal with Unable to find python module. but then it stops and runs the script anyways.
The entirely of ggml-gpt4all-j-v1.3-groovy is downloaded. However, this is the output it makes:
Found model file.
gptj_model_load: loading model from 'C:\\\\Users\\\\USER\\\\.cache\\\\gpt4all\\ggml-gpt4all-j-v1.3-groovy.bin' - please wait ...
gptj_model_load: n_vocab = 50400
gptj_model_load: n_ctx = 2048
gptj_model_load: n_embd = 4096
gptj_model_load: n_head = 16
gptj_model_load: n_layer = 28
gptj_model_load: n_rot = 64
gptj_model_load: f16 = 2
gptj_model_load: ggml ctx size = 5401.45 MB
gptj_model_load: kv self size = 896.00 MB
gptj_model_load: ................................... done
gptj_model_load: model size = 3609.38 MB / num tensors = 285
### Instruction:
The prompt below is a question to answer, a task to complete, or a conversation
to respond to; decide which and write an appropriate response.
### Prompt:
Name 3 colors
### Response:So it looks like the model is just giving a blank response? Not sure, that's all it outputs.
Also, I'm now using a direct copy paste of the example script, unlike the one I said before:
from gpt4all import GPT4All
gptj = GPT4All("ggml-gpt4all-j-v1.3-groovy")
messages = [{"role": "user", "content": "Name 3 colors"}]
gptj.chat_completion(messages)I get this warning:
[{
"resource": "/C:/Users/USER/Desktop/test.py",
"owner": "_generated_diagnostic_collection_name_#1",
"code": {
"value": "reportMissingImports",
"target": {
"$mid": 1,
"path": "/microsoft/pyright/blob/main/docs/configuration.md",
"scheme": "https",
"authority": "github.com",
"fragment": "reportMissingImports"
}
},
"severity": 4,
"message": "Import \"gpt4all\" could not be resolved",
"source": "Pylance",
"startLineNumber": 1,
"startColumn": 6,
"endLineNumber": 1,
"endColumn": 13
}]pylance is the thingie that helps with IDE stuff. I suppose it doesn't quite know what to do with the MSYS2 environment, or more specifically, the pip package of the gpt4all bindings.
As for the output: Looks good, but yes it should also produce something model generated. Although sometimes the model doesn't have anything to say. Still, it would be weird if it didn't in this case. Did you retry? I really hope it's not another problem.
Maybe also try running the example from the MINGW console to make sure that at least works.
gavtography
commented
1 year ago
pylanceis the thingie that helps with IDE stuff. I suppose it doesn't quite know what to do with the MSYS2 environment, or more specifically, the pip package of the gpt4all bindings.As for the output: Looks good, but yes it should also produce something model generated. Although sometimes the model doesn't have anything to say. Still, it would be weird if it didn't in this case. Did you retry? I really hope it's not another problem.
Maybe also try running the example from the MINGW console to make sure that at least works.
I won't have a chance to try running from the console for awhile, currently writing this on mobile. But in the meantime I can say that I did try running the script multiple times, every time gave an empty response from the model. I even tried changing the input from "Name 3 colors" to "Name 3 countries", still the same result.
Could be another issue unfortunately. I'll edit this response or make another reply sometime today when I get a chance.
cosmic-snow
commented
1 year ago In any case, you could try going back to the original Python interpreter you had that was already working fine in your environment. What you need to keep in mind/take care of:
pip install -e <path-to-gpt4all-bindings/python>, using the pip of the specific interpreter you want to use. That will make sure the bindings are available to that interpreter.gavtography
commented
1 year ago In any case, you could try going back to the original Python interpreter you had that was already working fine in your environment. What you need to keep in mind/take care of:
- run
pip install -e <path-to-gpt4all-bindings/python>, using thepipof the specific interpreter you want to use. That will make sure the bindings are available to that interpreter.- The DLLs you compiled are now native Windows DLLs, so they should work with a different interpreter.
- You have to make sure that their dependencies are also visible to that interpreter, as mentioned.
Alright so I went ahead and tried setting everything up for the original Python interpreter, but unfortunately I'm at the same point as before. The model is just outputting a blank response, no matter in the input.
Perhaps I need a bit more elaboration on "You have to make sure that their dependencies are also visible to that interpreter, as mentioned."? While I don't believe I'm missing anything, I'm confused on how I'm getting the same result, regardless of the interpreter.
Also, I tried running the script directly from the MINGW console as you suggested. But no dice, still gives a blank output. Makes me worried there's a bigger issue or something I completely missed.
Thanks again!
cosmic-snow
commented
1 year ago Perhaps I need a bit more elaboration on "You have to make sure that their dependencies are also visible to that interpreter, as mentioned."? While I don't believe I'm missing anything, I'm confused on how I'm getting the same result, regardless of the interpreter.
You're not missing anything. You've made it past the error FileNotFoundError: Could not find module ... which you would get if the interpreter was unable to pick up the dependencies (see new comment on that in https://github.com/nomic-ai/gpt4all/issues/710#issuecomment-1563337432, btw).
Also, I tried running the script directly from the MINGW console as you suggested. But no dice, still gives a blank output. Makes me worried there's a bigger issue or something I completely missed.
That's a "good" sign in that it's not a problem with your Visual Studio setup, although I'm not sure right now why it wouldn't work at this point. Maybe try doing a file integrity check on your model? In a MinGW console you should have an md5sum tool.
md5sum <path-to-model/ggml-gpt4all-j-v1.3-groovy.bin>It should output: 81a09a0ddf89690372fc296ff7f625af (as can be looked up here: https://gpt4all.io/models/models.json)
You can also try downloading a different model, e.g. ggml-mpt-7b-chat.bin (which then should have the checksum 756249d3d6abe23bde3b1ae272628640).
If nothing helps, I guess I'm going to try here with a different Python version to see if that's the culprit.
gavtography
commented
1 year ago Also, I tried running the script directly from the MINGW console as you suggested. But no dice, still gives a blank output. Makes me worried there's a bigger issue or something I completely missed.
That's a "good" sign in that it's not a problem with your Visual Studio setup, although I'm not sure right now why it wouldn't work at this point. Maybe try doing a file integrity check on your model? In a MinGW console you should have an
md5sumtool.md5sum <path-to-model/ggml-gpt4all-j-v1.3-groovy.bin>It should output:
81a09a0ddf89690372fc296ff7f625af(as can be looked up here: https://gpt4all.io/models/models.json) You can also try downloading a different model, e.g.ggml-mpt-7b-chat.bin(which then should have the checksum756249d3d6abe23bde3b1ae272628640).If nothing helps, I guess I'm going to try here with a different Python version to see if that's the culprit.
Alright so I did md5sum to the path of ggml-gpt4all-j-v1.3-groovy.bin and I got 81a09a0ddf89690372fc296ff7f625af so I believe that's good.
I also tried a different model as you suggested. However, ggml-mpt-7b-chat.bin gave this output:
Model downloaded at: C:\\\\Users\\\\USER\\\\.cache\\\\gpt4all\\ggml-mpt-7b-chat.bin
mpt_model_load: loading model from 'C:\\\\Users\\\\USER\\\\.cache\\\\gpt4all\\ggml-mpt-7b-chat.bin' - please wait ...
mpt_model_load: n_vocab = 50432
mpt_model_load: n_ctx = 2048
mpt_model_load: n_embd = 4096
mpt_model_load: n_head = 32
mpt_model_load: n_layer = 32
mpt_model_load: alibi_bias_max = 8.000000
mpt_model_load: clip_qkv = 0.000000
mpt_model_load: ftype = 2
mpt_model_load: ggml ctx size = 5653.09 MB
mpt_model_load: kv self size = 1024.00 MB
mpt_model_load: ........................ done
mpt_model_load: model size = 4629.02 MB / num tensors = 194
### Instruction:
The prompt below is a question to answer, a task to complete, or a conversation
to respond to; decide which and write an appropriate response.
### Prompt:
Hello, how are you?
### Response:
Assertion failed: ggml_nelements(src1) == 2, file C:/msys64/home/USER/gpt4all/gpt4all-backend/llama.cpp/ggml.c, line 9342uh, oh. that can be summed up as:
# Prompt:
Hello, how are you?
### Response:
not well at all.Sadly, I'm not very familiar with C++ backend of GPT4All and not at all familiar with the C++ backend that is llama.cpp (a separate project on which GPT4All builds).
What's the version of the repository you're using? (run git log and see at the top where it also says HEAD -> main gives a commit ID and comment)
Another thing to consider: are you certain you started fresh, i.e. without any lingering files from when you tried with MSVC before, when you compiled everything again with MinGW?
gavtography
commented
1 year ago uh, oh. that can be summed up as:
# Prompt: Hello, how are you? ### Response: not well at all.Sadly, I'm not very familiar with C++ backend of GPT4All and not at all familiar with the C++ backend that is
llama.cpp(a separate project on which GPT4All builds).What's the version of the repository you're using? (run
git logand see at the top where it also saysHEAD -> maingives a commit ID and comment)Another thing to consider: are you certain you started fresh, i.e. without any lingering files from when you tried with MSVC before, when you compiled everything again with MinGW?
Embarrassingly enough, I have absolutely no idea whatsoever where the git repository is located on my drive or what to even look up to locate it, so I can't run git log on it. Regardless, I first cloned it yesterday (May 24th). My lack of knowledge is probably starting to show 😅
And no, I cannot be certain that I did start fresh. I mean, I had no intention of going this in-depth with it and attempting to troubleshoot everything, so I'm really just trying to follow steps and advice, one thing at a time. I may or may not have left lingering files, honestly no clue, any way to verify if I did?
It seems like no matter what model I do, it results in a blank response or an error as the response.
cosmic-snow
commented
1 year ago Yes, this turned out to be very unfortunate overall, quite a string of problems. There's certainly a lot of room for improvement with the tooling.
The error said: C:/msys64/home/USER/gpt4all/gpt4all-backend/llama.cpp/ggml.c so I'm assuming the git repository is in C:\msys64\home\USER\gpt4all
But May 24th already narrows it down. I'll have to go look into what changed since my version, although that is May 22nd, so not much older.
To get back into a completely fresh state without any modifications you could run git clean -df. Although caution with that command, all your modifications (files and changes) will be lost, I believe, and it will be again as if you just cloned it. Then you'd have to run everything up to this point again.
Not sure about cmake itself. Looking at the help output, maybe running cmake --fresh .. in the build directory, then cmake --build . --parallel and all the other commands again would be enough.
Finally, in case it's really not a problem with the compilation but something else: you could try downloading and running the official installer for the chat application. That should work out of the box.
gavtography
commented
1 year ago Yes, this turned out to be very unfortunate overall, quite a string of problems. There's certainly a lot of room for improvement with the tooling.
The error said:
C:/msys64/home/USER/gpt4all/gpt4all-backend/llama.cpp/ggml.cso I'm assuming the git repository is inC:\msys64\home\USER\gpt4allBut May 24th already narrows it down. I'll have to go look into what changed since my version, although that is May 22nd, so not much older.
To get back into a completely fresh state without any modifications you could run
git clean -df. Although caution with that command, all your modifications (files and changes) will be lost, I believe, and it will be again as if you just cloned it. Then you'd have to run everything up to this point again.Not sure about
cmakeitself. Looking at the help output, maybe runningcmake --fresh ..in the build directory, thencmake --build . --paralleland all the other commands again would be enough.Finally, in case it's really not a problem with the compilation but something else: you could try downloading and running the official installer for the chat application. That should work out of the box.
Gotcha, well I went from scratch and tried a clean, fresh slate. This time ggml-mpt-7b-chat is giving a blank output instead of the error, but still, it's a blank output. ggml-gpt4all-j-v1.3-groovy also just gives a blank output.
I'm aware of the chat application, that works well, I'm more interested in playing around with it in Python specifically which is why I'm here. But if you're simply asking if the chat application functions, yes, no issues at all there.
I mean it quite simply seems to just not work with what I got here, wish I could say more unfortunately.
cosmic-snow
commented
1 year ago Ah so the chat works fine?
I'm really not sure what the problem is right at this moment and I guess I'd have to start from a clean slate here, too, and work through it and hope it fails like it does for you so I can troubleshoot the problem better. (Something for later.)
However, the official chat also has all the required DLLs. The Python bindings themselves are using ctypes, which means they dynamically interact with the native DLLs, i.e. they're not compiled against them. So you could use any for that. All that's really required is for Python to "see" the DLLs and the pip install step that makes the bindings package available to Python.
So what I'm saying is you could try with the chat's DLLs instead if all else fails. I don't have the time to test that step by step right now to then give instructions, but maybe you can piece it together yourself. It's basically about the 5 libraries: libllmodel.dll, libllama.dll, libstdc++-6.dll, libgcc_s_seh-1.dll, libwinpthread-1.dll (if they were compiled the same way for the chat). The Python interpreter you want to use needs access to working versions of all of those (python.exe folder or System32 folder are ugly but effective ways to ensure that).
gavtography
commented
1 year ago Ah so the chat works fine?
I'm really not sure what the problem is right at this moment and I guess I'd have to start from a clean slate here, too, and work through it and hope it fails like it does for you so I can troubleshoot the problem better. (Something for later.)
However, the official chat also has all the required DLLs. The Python bindings themselves are using
ctypes, which means they dynamically interact with the native DLLs, i.e. they're not compiled against them. So you could use any for that. All that's really required is for Python to "see" the DLLs and thepip installstep that makes the bindings package available to Python.So what I'm saying is you could try with the chat's DLLs instead if all else fails. I don't have the time to test that step by step right now to then give instructions, but maybe you can piece it together yourself. It's basically about the 5 libraries:
libllmodel.dll,libllama.dll,libstdc++-6.dll,libgcc_s_seh-1.dll,libwinpthread-1.dll(if they were compiled the same way for the chat). The Python interpreter you want to use needs access to working versions of all of those (python.exe folder or System32 folder are ugly but effective ways to ensure that).
Chat is working fine. But if you're saying that these 5 files libllmodel.dll, libllama.dll, libstdc++-6.dll, libgcc_s_seh-1.dll, libwinpthread-1.dll are somewhere in C:\Users\USER\gpt4all, then I can't seem to find them.
cosmic-snow
commented
1 year ago No, I meant the chat application's versions, wherever you have that installed. It should have all of those, too.
gavtography
commented
1 year ago No, I meant the chat application's versions, wherever you have that installed. It should have all of those, too.
Right, C:\Users\USER\gpt4all is the file path to the chat application. So the dll files should be in there is what you're saying? I've redownloaded the chat application, right clicked the shortcut and opened the file location, which is how I discovered that directory. I tried searching one by one for those dll files but none of them exist in that directory. Apologies if I'm completely misunderstanding, perhaps I need an oversimplified explanation if I'm mistaken.
cosmic-snow
commented
1 year ago They're here for me:
but I just saw in GitHub's version history, they changed some things, so maybe after the update today they don't ship these versions anymore. 😓
gavtography
commented
1 year ago but I just saw in GitHub's version history, they changed some things, so maybe after the update today they don't ship these versions anymore. 😓
Are the files renamed possibly? I noticed a lib folder that has llama.dll and llmodel.dll opposed to "libllama.dll" and "libllmodel.dll"
Edit: well I see those files in your screenshot too, so perhaps they're something different. Pretty stuck then.
cosmic-snow
commented
1 year ago Not just that, different compiler:
Might need a fix to the pip install step. Not going to test that now. Maybe just wait a day or two now, in case something else needs to be changed for the Python bindings.
cosmic-snow
commented
1 year ago Are the files renamed possibly? I noticed a lib folder that has llama.dll and llmodel.dll opposed to "libllama.dll" and "libllmodel.dll"
Edit: well I see those files in your screenshot too, so perhaps they're something different. Pretty stuck then.
Alright, so, remember when I said:
The Python bindings themselves are using ctypes, which means they dynamically interact with the native DLLs, i.e. they're not compiled against them. So you could use any for that.
I've tested it now with the new llama.dll and llmodel.dll and got it to work. In one way it's even simpler (at least on my system), but it requires editing the Python bindings. I'm not sure if you want to attempt that, but here are some instructions anyway. These instructions rely on existing DLLs from the chat application, not self-compiled ones:
You might want to do a
Now to avoid messing with existing things a lot, here's a (maybe new) concept: Python virtual environments.git pull and clean your repository, but that's optional, the Python binding code hasn't changed (yet).
First of all, you would decide which Python interpreter you're going to use.
There's a minor difference between the Windows installed Python and the MSYS2 Python. Windows installed Python typically creates a Scripts/ folder in the virtual environment, otherwise you typically have a bin/ folder for the native programs and libraries. To keep it simple, the following instructions here are again for when you're doing everything in a MinGW console (and not using the Windows installed Python):
python -m venv venv # invoke the venv module, create a venv/ folder which contains the virtual environment
source venv/bin/activate # this console session now uses the virtual environment
# to check:
type -p python pip # should print: '<full-path-to>/venv/bin/python' and '<full-path-to>/venv/bin/pip'
pip install -e <path-to-gpt4all-repository>/gpt4all-bindings/python # installs the 'gpt4all' bindings package in the virtual environmentNow copy the llama.dll and llmodel.dll from your chat application into the venv/bin folder, so they're right next to your virtual environment's python.exe. Then here is where it might be simpler with these DLLs. Your system should already know where the dependencies are. To check, run:
ldd venv/bin/llama.dll
ldd venv/bin/llmodel.dllIf you don't see a => not found in the output of these two commands, you're good to go.
Now in order to make the Python bindings use these and not the ones you'd otherwise compile, you need to edit the pyllmodel.py file of the Python bindings, it's in <path-to-repository>/gpt4all-bindings/python/gpt4all/pyllmodel.py. There are two things to do:
pkg_resources to find the libraries (they're already next to the Python interpreter)So comment out the pkg_resources lines and simply put in the right DLL names where they're loaded with ctypes.CDLL(...). Also comment out the next two llama_dir/llmodel_dir = ... because they're referencing themselves but aren't set:
diff --git a/gpt4all-bindings/python/gpt4all/pyllmodel.py b/gpt4all-bindings/python/gpt4all/pyllmodel.py
index 6117c9f..930ec38 100644
--- a/gpt4all-bindings/python/gpt4all/pyllmodel.py
+++ b/gpt4all-bindings/python/gpt4all/pyllmodel.py
@@ -39,15 +39,17 @@ def load_llmodel_library():
llmodel_file = "libllmodel" + '.' + c_lib_ext
llama_file = "libllama" + '.' + c_lib_ext
- llama_dir = str(pkg_resources.resource_filename('gpt4all', os.path.join(LLMODEL_PATH, llama_file)))
- llmodel_dir = str(pkg_resources.resource_filename('gpt4all', os.path.join(LLMODEL_PATH, llmodel_file)))
+ #llama_dir = str(pkg_resources.resource_filename('gpt4all', os.path.join(LLMODEL_PATH, llama_file)))
+ #llmodel_dir = str(pkg_resources.resource_filename('gpt4all', os.path.join(LLMODEL_PATH, llmodel_file)))
# For windows
- llama_dir = llama_dir.replace("\\", "\\\\")
- llmodel_dir = llmodel_dir.replace("\\", "\\\\")
+ #llama_dir = llama_dir.replace("\\", "\\\\")
+ #llmodel_dir = llmodel_dir.replace("\\", "\\\\")
- llama_lib = ctypes.CDLL(llama_dir, mode=ctypes.RTLD_GLOBAL)
- llmodel_lib = ctypes.CDLL(llmodel_dir)
+ #llama_lib = ctypes.CDLL(llama_dir, mode=ctypes.RTLD_GLOBAL)
+ #llmodel_lib = ctypes.CDLL(llmodel_dir)
+ llama_lib = ctypes.CDLL('llama.dll', mode=ctypes.RTLD_GLOBAL)
+ llmodel_lib = ctypes.CDLL('llmodel.dll')
return llmodel_lib, llama_libIt then looks like this:
llmodel_file = "libllmodel" + '.' + c_lib_ext
llama_file = "libllama" + '.' + c_lib_ext
#llama_dir = str(pkg_resources.resource_filename('gpt4all', os.path.join(LLMODEL_PATH, llama_file)))
#llmodel_dir = str(pkg_resources.resource_filename('gpt4all', os.path.join(LLMODEL_PATH, llmodel_file)))
# For windows
#llama_dir = llama_dir.replace("\\", "\\\\")
#llmodel_dir = llmodel_dir.replace("\\", "\\\\")
#llama_lib = ctypes.CDLL(llama_dir, mode=ctypes.RTLD_GLOBAL)
#llmodel_lib = ctypes.CDLL(llmodel_dir)
llama_lib = ctypes.CDLL('llama.dll', mode=ctypes.RTLD_GLOBAL)
llmodel_lib = ctypes.CDLL('llmodel.dll')
return llmodel_lib, llama_libNow with all that done -- and hopefully I didn't make any mistakes when writing the instructions -- you should be able to run an example from within that console.
To target a virtual environment from within Visual Studio, point it to virtual environment's python.exe. It should know how to handle those.
gavtography
commented
1 year ago Funnily enough, the solution was simply uninstalling gpt4all using pip and reinstalling it. Of course, this only worked after I followed all your instructions to get this working properly. Sweet!
Definitely a large string of issues that led to this point though. Ton of troubleshooting, but can confirm it technically is working with my setup, so I suppose that's good enough to close the thread because the title is no longer true.
I appreciate the time and effort to getting to the bottom of this.
cosmic-snow
commented
1 year ago @gavtography that's great to hear you got it to work after all. (pinging in hopes you'll see it even after closing, sorry for that)
I do still wonder how it got into a state where it'd produce these kinds of errors. Pretty unlucky -- you kind of got thrown into the deep end of the pool on that.
On the bright side, you now know quite a bit more about the Python <-> native interaction on Windows. You should from now on technically be able to interface with any native DLL that exposes a C interface, once you look into ctypes a bit more.
Getting things to work was the first priority, here are a few more thoughts:
clean-up:
If you copied any DLLs into places while trying to figure things out and they are not required anymore for it to work (esp. into System32), remove them again. You never know when you might run into subtle versioning problems at some later point.
Similar goes for Python's packages. If the gpt4all package or one of its dependencies is now installed where you don't need it (I mean a different interpreter), remove it again.
Myself, I've made it a habit to pretty much always work with virtual environments for my own stuff. It's definitely worth looking into that (if you're not familiar already, of course).
tools:
for troubleshooting these kinds of things, some of GNU tools are neat (such as the mentioned ldd), but there are also some Windows tools I can recommend. Especially: Windows Sysinternals and Dependency Walker -- both are free to use.
I'm assuming that in the future, the Windows parts will rely solely on the MSVC compiler. And you already had Cygwin, so MSYS2 is kind of obsolete now (but keep it if you like it).
With that, I'd also recommend to then use the "normal" Python installation on Windows -- at least once the devs publish a working bindings package to PyPI (there already is one now, but it doesn't have the DLLs). It's just the "path more well-treaded".
I appreciate the time and effort to getting to the bottom of this.
You're welcome. I was thinking about maybe compressing it into some kind of guide once it's resolved, but looks like that soon won't be needed anymore, anyway. However, you're likely not the only one to run into (at least some of) these problems, so as detailed as this exchange was, it probably helps some others, too. (just have to trust the search engines 😉)
gavtography
commented
1 year ago No worries, I get email notifications every time someone replies to this issue anyway. From my basic understanding, my best guess is that I started on the wrong foot by trying to use Cygwin instead of MSYS2 MINGW64, then I continued that wrong foot by using the incorrect terminal (MSYS2 MSYS instead of MSYS2 MINGW64), must have gotten a bunch of files mixed up between terminals + using the incorrect python interpreter.
Ultimately, I think your advice of doing a completely fresh install of everything (on the correct terminal) + me deciding to reinstall gpt4all using pip was the solution. Although, I did update windows last night for the first time in awhile, but I'll assume that's a coincidence.
But you're completely right, I think I've learned a lot from this process, so that's a good takeaway. I'm sure this thread will be helpful to others too, but maybe there should be a little more documentation on the python bindings, it wouldn't hurt.
cosmic-snow
commented
1 year ago I'm sure this thread will be helpful to others too, but maybe there should be a little more documentation on the python bindings, it wouldn't hurt.
I certainly agree with that. But maybe not on the MinGW process.
Once the whole "doing it with MSVC" is ready, it will be much less problematic in general. If you run ldd on e.g. the new llmodel.dll in v2.4.4 of the chat application (or use the Dependency Walker), it should already know where its dependencies are. For example, the C++ runtime library should be part of the OS itself (if not, there's a ready-made installer provided by Microsoft).
gavtography
commented
1 year ago @cosmic-snow Sorry for the ping, not sure if you'd see this without it. I figured I'd ask you this question instead of making a new post, especially since it isn't technically an issue. Hoping you have some information about this.
One thing I don't quite understand about the python bindings is how memory works. For example if I ask the language model "What is orange juice?", it gives the proper response, but then if I ask "can you elaborate on that?" then of course it has no idea what I'm talking about because it can't refer to previous messages. I don't quite understand how to implement this or if it's even possible. I can see this exists in the chat client, but again the python bindings documentation is so barebones that I don't really have much to go off of.
I have this large python script with a bunch of other stuff in it, but inside it I included this little AOL-messenger styled chat where I can type a message, then the language model responds. I'd prefer if the language model had some type of memory or context. I suppose I could technically create a little workaround system to have some type of short-term memory now that I think about it, but I'm wondering if there's some sort of official implementation of this already in the python library.
Also, I always see this in the python terminal when I run the code:
### Instruction:
The prompt below is a question to answer, a task to complete, or a conversation
to respond to; decide which and write an appropriate response.How do I edit these instructions to be whatever I want?
I appreciate the help, thanks!
cosmic-snow
commented
1 year ago One thing I don't quite understand about the python bindings is how memory works. For example if I ask the language model "What is orange juice?", it gives the proper response, but then if I ask "can you elaborate on that?" then of course it has no idea what I'm talking about because it can't refer to previous messages.
I'm kind of new to the whole thing myself, but I can tell you: unless you're restarting your program, that's not what happens. In fact, these models have "perfect memory" as long as it fits into their context, which is typically 2048 tokens. But sometimes they're just not very good at that (recall/reasoning/confusion). I've started looking into how to improve that myself through better prompting here. In the chat application, you can try regenerating the answer, with the Python bindings, you'd have to program that out yourself. You can also try reformulating your prompt, that might help.
I'd prefer if the language model had some type of memory or context. I suppose I could technically create a little workaround system to have some type of short-term memory now that I think about it, but I'm wondering if there's some sort of official implementation of this already in the python library.
Any input you give it is context. Simply prepend it to the first time you call the model but don't show that on screen. You could try manipulating header and footer (mentioned below), but they're only True/False arguments inchat_completion() so that's not enough. You could e.g. create a subclass and override _build_prompt() with your own logic.
How do I edit these instructions to be whatever I want?
gpt4all.GPT4All is the main class of the bindings. From within a Python console, you can do:
from gpt4all import GPT4All
help(GPT4All.chat_completion)Alternatively, have a look at the source: https://github.com/nomic-ai/gpt4all/blob/7aed367c68b7b5c5ade2f5d4358fa8d9d865ccfd/gpt4all-bindings/python/gpt4all/gpt4all.py#L175-L181
You can change the default_prompt_header and default_prompt_footer arguments. Or override some things or write your own - I'm not too big of a fan of how that works there. 😉
I'm a complete beginner, so apologies if I'm missing something obvious. I'm trying to follow the README here https://github.com/nomic-ai/gpt4all/blob/main/gpt4all-bindings/python/README.md
I install gpt4all with pip perfectly fine. Then I installed the Cygwin64 Terminal and ran the lines in the tutorial. Everything goes well until "cmake --build . --parallel". This is what I get:
From what I can see, some type of error is happening. Especially because "libllmodel.*" does not exist in "gpt4all-backend/build".
If I continue the tutorial anyway and try to run the python code. Then "pyllmodel.py" opens on Visual Studio and I get the following error:
Not sure if it's a bug, or I'm missing something, but any help would be appreciated. Reminder that I'm a beginner, so hoping for not too much technical jargon that might be difficult for me to understand. Thanks!
Information
Related Components
Reproduction
Simply following the README on Windows 11, Python 3.9. Nothing special.
Expected behavior
For the example python script to successfully output the result to "Name 3 colors" after downloading "ggml-gpt4all-j-v1.3-groovy".