Incompatibility between GPT4All and Vosk speech recognizer.

[karencfisher]

System Info

I am testing GPT4All in a voice chat application. I am using the Vosk speech recognition package to convert speech to text. However, if I import gpt4all, and after that vosk:

import gpt4all
from vosk_recognizer import SpeechRecognize

my vosk_recognizer module in turn imports:

import vosk

I get this error:

File "C:\Users\karen\myprojects\simple-chat\ChatGPT.py", line 21, in <module>
    from vosk_recognizer import SpeechRecognize
  File "C:\Users\karen\myprojects\simple-chat\vosk_recognizer.py", line 7, in <module>
    import vosk
  File "C:\Users\karen\myprojects\simple-chat\chat_env\lib\site-packages\vosk\__init__.py", line 36, in <module>
    _c = open_dll()
  File "C:\Users\karen\myprojects\simple-chat\chat_env\lib\site-packages\vosk\__init__.py", line 28, in open_dll
    return _ffi.dlopen(os.path.join(dlldir, "libvosk.dll"))
OSError: cannot load library 'C:\Users\karen\myprojects\simple-chat\chat_env\lib\site-packages\vosk\libvosk.dll': error 0x7f  

If I reverse the order of imports (import vosk and then gpt4all), I get an error importing gpt4all:

File "C:\Users\karen\myprojects\simple-chat\ChatGPT.py", line 24, in <module>
    import gpt4all
  File "C:\Users\karen\myprojects\simple-chat\chat_env\lib\site-packages\gpt4all\__init__.py", line 1, in <module>
    from .pyllmodel import LLModel # noqa
  File "C:\Users\karen\myprojects\simple-chat\chat_env\lib\site-packages\gpt4all\pyllmodel.py", line 49, in <module>
    llmodel = load_llmodel_library()
  File "C:\Users\karen\myprojects\simple-chat\chat_env\lib\site-packages\gpt4all\pyllmodel.py", line 45, in load_llmodel_library    llmodel_lib = ctypes.CDLL(llmodel_dir)
  File "C:\Users\karen\Anaconda3\lib\ctypes\__init__.py", line 381, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 127] The specified procedure could not be found

Name space conflict? This is a show stopper to implement gpt4all in this project. :(

Information

• [ ] The official example notebooks/scripts
• [X] My own modified scripts

Related Components

• [ ] backend
• [ ] bindings
• [x] python-bindings
• [ ] chat-ui
• [ ] models
• [ ] circleci
• [ ] docker
• [ ] api

Reproduction

1) import gpt4all 2) import vosk

Or vice versa to produce second error reported above.

Expected behavior

No conflict. Happy family. ;)

[cosmic-snow]

Interesting. I can at least confirm this for now with:

• gpt4all v0.3.4
• vosk v0.3.45

I haven't really investigated it yet, though.

[cosmic-snow]

Alright, my first guess is: Since it looks like they've both been built with some form of MinGW and both ship their own C++ runtimes, these somehow get in each others' way.

When I try to import vosk before gpt4all I get an error popup complaining about a "procedure entry point" that does not actually exist with that exact name in the DLL it looks for it, although a similarly name-mangled one does.

I think one solution would be to compile both packages on your system with the exact same toolchain. Another would be to try building gpt4all with MSVC instead while still retrieving vosk from PyPI. That would require a bit of extra work, though.

I might try some other things later.

[hy-atharv]

Hey, I am trying to achieve the same too and facing the exactly same errors. I finally somehow found the conversation related to this issue. Are there any updates on it? Any solutions? I am really in need of this.

[hy-atharv]

Alright, my first guess is: Since it looks like they've both been built with some form of MinGW and both ship their own C++ runtimes, these somehow get in each others' way.

When I try to import vosk before gpt4all I get an error popup complaining about a "procedure entry point" that does not actually exist with that exact name in the DLL it looks for it, although a similarly name-mangled one does.

I think one solution would be to compile both packages on your system with the exact same toolchain. Another would be to try building gpt4all with MSVC instead while still retrieving vosk from PyPI. That would require a bit of extra work, though.

I might try some other things later.

Can you try and tell me if any of the method working?, although i myself tried to compile them, but i was facing some errors in cmake and paths. I really need help in this. If its working, it would really be helpful for me. I have been facing this incompatibility issue for days. I am really in need of help for this.

[cosmic-snow]

Can you try and tell me if any of the method working?

Yes, at least one of them is confirmed to be working (MSVC), but there's no reason the other wouldn't. Walked someone through it on the Discord.

I can't tell you how to solve your build problems if you don't say what they are.

[hy-atharv]

alr mate, I will try the MSVC method once. umm can you pls tell me the steps to do that on windows? so that it can be clear to me once with no confusion that I may have when i will be doing without knowing.

[cosmic-snow]

First of all, read the repository readme on how to build the Python bindings carefully.

Here's another version of the steps:

• Install Visual Studio or the C/C++ build tools.
• Make sure you have a version of cmake somewhere. VS comes with one.
• Start a console through vcvars64.bat or vcvarsall.bat. It's definitely included with VS, not sure about the build tools, but probably.
• Clone repository with --recurse-submodules or run after clone: git submodule update --init.
• cd to gpt4all-backend
• Run:

  md build
  cd build
  cmake ..

• After that there's a .sln solution file in that repository. you can build that with either cmake (cmake --build . --parallel --config Release) or open and build it in VS.
• Create a Python virtual environment and activate it.
• pip install -e gpt4all\gpt4all-bindings\python
• It'll copy the DLLs to a subfolder of the Python bindings folder (something like ...DO_NOT_MODIFY).
• In a subfolder you'll have a llmodel.dll. Either adjust the bindings code so it won't look for a libllmodel.dll or rename that to libllmodel.dll.
• Copy and run an example to test just the bindings.

[hy-atharv]

First of all, read the repository readme on how to build the Python bindings carefully.

Here's another version of the steps:

• Install Visual Studio or the C/C++ build tools.
• Make sure you have a version of cmake somewhere. VS comes with one.
• Start a console through vcvars64.bat or vcvarsall.bat. It's definitely included with VS, not sure about the build tools, but probably.
• Clone repository with --recurse-submodules or run after clone: git submodule update --init.
• cd to gpt4all-backend
• Run:

  md build
  cd build
  cmake ..

• After that there's a .sln solution file in that repository. you can build that with either cmake (cmake --build . --parallel --config Release) or open and build it in VS.
• Create a Python virtual environment and activate it.
• pip install -e gpt4all\gpt4all-bindings\python
• It'll copy the DLLs to a subfolder of the Python bindings folder (something like ...DO_NOT_MODIFY).
• In a subfolder you'll have a llmodel.dll. Either adjust the bindings code so it won't look for a libllmodel.dll or rename that to libllmodel.dll.
• Copy and run an example to test just the bindings.

Alright so I performed all the steps and it was built. But about the step creating a virtual environment and activating it, I couldnt do it in my terminal, so I did that in my pycharm project folder. There I created a virtual environment, activated it and then did pip install -e (path to python bindings). And then went to do_not_modify folder and in a subfolder, renamed llmodel .dll to libllmodel.dll. And then I made python file test and there i copied the testing code for gpt4all, and then the import was not happening because it said gpt4all package isnt installed, so i installed gpt4 package by pip install. and then file ran. but again as i went to my another file where I am trying to import both vosk and gpt4all, the same issue persisted. I think I maybe did something wrong. the built was successful as it seemed to me. Maybe i messed up in virtual environment steps. Can you tell me the steps after build in detail. Maybe i will do it again.

[cosmic-snow]

... And then I made python file test and there i copied the testing code for gpt4all, and then the import was not happening because it said gpt4all package isnt installed, so i installed gpt4 package by pip install. and then file ran. but again as i went to my another file where I am trying to import both vosk and gpt4all, the same issue persisted. I think I maybe did something wrong.

Yes, probably. You went wrong somewhere before it said it's not installed. Doing pip install gpt4all afterwards installed the online version, not the locally built one is what I assume. Make sure you have the virtualenv activated when doing pip install -e ... and make sure you have it activated when running the test.

Also, check the Python documentation, look for venv to figure out how to create one in the console, if necessary.

[hy-atharv]

... And then I made python file test and there i copied the testing code for gpt4all, and then the import was not happening because it said gpt4all package isnt installed, so i installed gpt4 package by pip install. and then file ran. but again as i went to my another file where I am trying to import both vosk and gpt4all, the same issue persisted. I think I maybe did something wrong.

Yes, probably. You went wrong somewhere before it said it's not installed. Doing pip install gpt4all afterwards installed the online version, not the locally built one is what I assume. Make sure you have the virtualenv activated when doing pip install -e ... and make sure you have it activated when running the test.

Also, check the Python documentation, look for venv to figure out how to create one in the console, if necessary.

Alright so first of all

THANK YOU SO MUCH!! YOU SOLVED BIGGEST PROBLEM OF MY LIFE!!!

Now coming on to the thing..

I repeated the whole process again and this time I deleted my old project folder in pycharm too, it was a virtual environment too. So, it had a lot of site packages installed and I thought it would be good to start again my project in the new virtual environment in which I ran the commmand pip install -e gpt4all\gpt4all-bindings\python in Command prompt. Now this time the virtual environment I created in command prompt and activated it. Then ran the command pip install -e gpt4all\gpt4all-bindings\python. Then yeah did other steps such as renaming the file llmodel.dll to libllmodel.dll and then after all the steps. I opened that virtual environment in my pycharm and then made a python file test. In that I tested the code for gpt4all, it was working and importing gpt4all, I didnt have to install the gpt4all package, and then I kept a hand on my heart and installed the vosk module by pip3 install vosk in that virtual environment and then I did the real thing - "Importing vosk and gpt4all together" for which this discussion has been going for - "Solving the incompatibility issue between gpt4all and vosk". Well the news is it WORKS!!. Vosk is retrieving from PyPI while gpt4all is locally built with MSVC And thats how they are not conflicting with each other now. Once again thank you so much @cosmic-snow for your great help and the time of yours you allocated for this issue!

[hy-atharv]

... And then I made python file test and there i copied the testing code for gpt4all, and then the import was not happening because it said gpt4all package isnt installed, so i installed gpt4 package by pip install. and then file ran. but again as i went to my another file where I am trying to import both vosk and gpt4all, the same issue persisted. I think I maybe did something wrong.

Yes, probably. You went wrong somewhere before it said it's not installed. Doing pip install gpt4all afterwards installed the online version, not the locally built one is what I assume. Make sure you have the virtualenv activated when doing pip install -e ... and make sure you have it activated when running the test. Also, check the Python documentation, look for venv to figure out how to create one in the console, if necessary.

Alright so first of all

THANK YOU SO MUCH!! YOU SOLVED BIGGEST PROBLEM OF MY LIFE!!!

Now coming on to the thing..

I repeated the whole process again and this time I deleted my old project folder in pycharm too, it was a virtual environment too. So, it had a lot of site packages installed and I thought it would be good to start again my project in the new virtual environment in which I ran the commmand pip install -e gpt4all\gpt4all-bindings\python in Command prompt. Now this time the virtual environment I created in command prompt and activated it. Then ran the command pip install -e gpt4all\gpt4all-bindings\python. Then yeah did other steps such as renaming the file llmodel.dll to libllmodel.dll and then after all the steps. I opened that virtual environment in my pycharm and then made a python file test. In that I tested the code for gpt4all, it was working and importing gpt4all, I didnt have to install the gpt4all package, and then I kept a hand on my heart and installed the vosk module by pip3 install vosk in that virtual environment and then I did the real thing - "Importing vosk and gpt4all together" for which this discussion has been going for - "Solving the incompatibility issue between gpt4all and vosk". Well the news is it WORKS!!. Vosk is retrieving from PyPI while gpt4all is locally built with MSVC And thats how they are not conflicting with each other now. Once again thank you so much @cosmic-snow for your great help and the time of yours you allocated for this issue!

@cosmic-snow , So they both can run together but I am actually having a speed problem with the gpt4all model, the speed has become really slow when I am using the model in this MSVC built library of gpt4all in a python environment, now it takes 2-3 minutes for a max 200 Tokens. Before when I used to just import it using PyPl, the model used to generate response a bit quicker.( around 30 secs). Can you tell me whats causing this and is there any workaround I could do to increase the speed of response generated by the gpt4all? Any way possible?

[JamieGrimwood]

Has there been any updates with this? This is still an issue!

[cosmic-snow]

Has there been any updates with this? This is still an issue!

If I remember correctly, the problem is pretty clear; there are two runtimes which are incompatible because two different, independent projects are relying on different versions of the runtime. There are two straight-forward solutions to that:

• Compile the native parts of these packages with your own toolchain on your own computer, then there won't be a mismatch anymore.

• Compile GPT4All's package on your own computer with the MSVC compiler as outlined above; then there won't be a mismatch because there are two independent runtimes which don't interfere with each other. If you read through the comments, this has been proven to work.

The problem here really is the inherent nature of the native, compiled parts of these packages. I guess it's still open because no one has come up with a way to deal with that in a friendlier way. The mentioned workaround should be fine, although note that, I haven't tried that in a while.

[cebtenzzre]

Probably adding RTLD_DEEPBIND to our dlopen flags would help. We already use RTLD_LOCAL, but our version of llama.cpp must be using symbols from other versions. Another option would be building llama.cpp with -Bsymbolic. That's all Linux stuff, though - not sure about how to solve this on Windows.