superbooga not installing properly

[2blackbar]

Describe the bug

ERROR:Failed to load the extension "superbooga". Traceback (most recent call last): File "F:\oobabooga-windows\text-generation-webui\modules\extensions.py", line 34, in load_extensions exec(f"import extensions.{name}.script") File "", line 1, in File "F:\oobabooga-windows\text-generation-webui\extensions\superbooga\script.py", line 10, in from .chromadb import add_chunks_to_collector, make_collector File "F:\oobabooga-windows\text-generation-webui\extensions\superbooga\chromadb.py", line 1, in import chromadb ModuleNotFoundError: No module named 'chromadb'

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

enable it

Screenshot

No response

Logs

ERROR:Failed to load the extension "superbooga".
Traceback (most recent call last):
  File "F:\oobabooga-windows\text-generation-webui\modules\extensions.py", line 34, in load_extensions
    exec(f"import extensions.{name}.script")
  File "<string>", line 1, in <module>
  File "F:\oobabooga-windows\text-generation-webui\extensions\superbooga\script.py", line 10, in <module>
    from .chromadb import add_chunks_to_collector, make_collector
  File "F:\oobabooga-windows\text-generation-webui\extensions\superbooga\chromadb.py", line 1, in <module>
    import chromadb
ModuleNotFoundError: No module named 'chromadb'

System Info

win10

[2blackbar]

looks like im not first, why preinstalled extensions dont have preinstalled requirements whats the point ? Im not a python dev but some devs are forcing it, do i have to add pull request with updated requirements from extensions really ? cmon guys, its not for me, its for non devs cause i already dealt with this If someone wants to fix on its own, activate enviroment, activate.bat is in conda/bin folder F:\oobabooga-windows\installer_files\conda\condabin

So, do it like this

F:\oobabooga-windows\installer_files\conda\condabin\activate.bat F:\oobabooga-windows\installer_files\env

obviousy change the paths to your own, then do - pip install chromadb==0.3.18

[toast22a]

I checked the one-click installer repo and apparently, it does install requirements for extensions, specifically except for superbooga: https://github.com/oobabooga/one-click-installers/blob/4ef2de348695ccdb616ad5a7d9b3540e59de9197/webui.py#L87-L96

Apparently it has something to do with hnswlib (a dependency of chromadb) installation not working right on Windows. It's covered in the issue here: https://github.com/oobabooga/one-click-installers/issues/56 .

If you'd like to make an issue/PR for this, it should probably go under the one-click-installers repo, not here.

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.

[Kippykip]

I still can't install this shit either

[TheLustriVA]

The extension's still not picking up my install of ChromaDB.

[revolvedai]

Same here, also the fix of running cmd_windows.bat and then running pip install chromadb was not working as the wheel still fails to build. The error message was: "Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/"

I installed the build tools (latest package with C++ support) and was still getting the error. By modifying the installation under Desktop C++ environment and installing MSVC v142 VS 2019, Windows 10 SDK (.20) and C++/CLI Support for v143 build tools I was able to install ChromaDB and build the wheel successfully. Superbooga loaded on my next run of Ooga.

[nixudos]

looks like im not first, why preinstalled extensions dont have preinstalled requirements whats the point ? Im not a python dev but some devs are forcing it, do i have to add pull request with updated requirements from extensions really ? cmon guys, its not for me, its for non devs cause i already dealt with this If someone wants to fix on its own, activate enviroment, activate.bat is in conda/bin folder F:\oobabooga-windows\installer_files\conda\condabin

So, do it like this

F:\oobabooga-windows\installer_files\conda\condabin\activate.bat F:\oobabooga-windows\installer_files\env

obviousy change the paths to your own, then do - pip install chromadb==0.3.18

This worked like a charm on my Windows 10. Thank you!

[AvidGameFan]

looks like im not first, why preinstalled extensions dont have preinstalled requirements whats the point ? Im not a python dev but some devs are forcing it, do i have to add pull request with updated requirements from extensions really ? cmon guys, its not for me, its for non devs cause i already dealt with this If someone wants to fix on its own, activate enviroment, activate.bat is in conda/bin folder F:\oobabooga-windows\installer_files\conda\condabin So, do it like this F:\oobabooga-windows\installer_files\conda\condabin\activate.bat F:\oobabooga-windows\installer_files\env obviousy change the paths to your own, then do - pip install chromadb==0.3.18

This worked like a charm on my Windows 10. Thank you!

Specifically, I ran "cmd_windows.bat", which sets up the environment. I installed the requirements.txt, but ran into the above-mentioned error. After finding this posting, I ran: pip install chromadb==0.3.18

It got further, but still had an error:

2023-10-22 22:43:30 ERROR:Failed to load the extension "superboogav2".
Traceback (most recent call last):
  File "D:\text-generation-webui\modules\extensions.py", line 36, in load_extensions
    exec(f"import extensions.{name}.script")
  File "<string>", line 1, in <module>
  File "D:\text-generation-webui\extensions\superboogav2\script.py", line 21, in <module>
    from .download_urls import feed_url_into_collector
  File "D:\text-generation-webui\extensions\superboogav2\download_urls.py", line 9, in <module>
    from .data_processor import process_and_add_to_collector
  File "D:\text-generation-webui\extensions\superboogav2\data_processor.py", line 12, in <module>
    from .data_preprocessor import TextPreprocessorBuilder, TextSummarizer
  File "D:\text-generation-webui\extensions\superboogav2\data_preprocessor.py", line 14, in <module>
    import pytextrank
ModuleNotFoundError: No module named 'pytextrank'

Ok, fine, I ran: pip install pytextrank Another error, then: pip install optuna

Ok, NOW I think it's happy!

[enzomtpYT]

looks like im not first, why preinstalled extensions dont have preinstalled requirements whats the point ? Im not a python dev but some devs are forcing it, do i have to add pull request with updated requirements from extensions really ? cmon guys, its not for me, its for non devs cause i already dealt with this If someone wants to fix on its own, activate enviroment, activate.bat is in conda/bin folder F:\oobabooga-windows\installer_files\conda\condabin So, do it like this F:\oobabooga-windows\installer_files\conda\condabin\activate.bat F:\oobabooga-windows\installer_files\env obviousy change the paths to your own, then do - pip install chromadb==0.3.18

This worked like a charm on my Windows 10. Thank you!

Specifically, I ran "cmd_windows.bat", which sets up the environment. I installed the requirements.txt, but ran into the above-mentioned error. After finding this posting, I ran: pip install chromadb==0.3.18

It got further, but still had an error:

2023-10-22 22:43:30 ERROR:Failed to load the extension "superboogav2".
Traceback (most recent call last):
  File "D:\text-generation-webui\modules\extensions.py", line 36, in load_extensions
    exec(f"import extensions.{name}.script")
  File "<string>", line 1, in <module>
  File "D:\text-generation-webui\extensions\superboogav2\script.py", line 21, in <module>
    from .download_urls import feed_url_into_collector
  File "D:\text-generation-webui\extensions\superboogav2\download_urls.py", line 9, in <module>
    from .data_processor import process_and_add_to_collector
  File "D:\text-generation-webui\extensions\superboogav2\data_processor.py", line 12, in <module>
    from .data_preprocessor import TextPreprocessorBuilder, TextSummarizer
  File "D:\text-generation-webui\extensions\superboogav2\data_preprocessor.py", line 14, in <module>
    import pytextrank
ModuleNotFoundError: No module named 'pytextrank'

Ok, fine, I ran: pip install pytextrank Another error, then: pip install optuna

Ok, NOW I think it's happy!

Yea i've had the same errors as you, but after unstalling optuna i get this error now...

2023-11-21 16:08:53 ERROR:Failed to load the extension "superboogav2". Traceback (most recent call last): File "D:\Programmes\oobabooga_windows\text-generation-webui-main\modules\extensions.py", line 41, in load_extensions extension.setup() File "D:\Programmes\oobabooga_windows\text-generation-webui-main\extensions\superboogav2\script.py", line 35, in setup collector = make_collector() ^^^^^^^^^^^^^^^^ File "D:\Programmes\oobabooga_windows\text-generation-webui-main\extensions\superboogav2\chromadb.py", line 376, in make_collector return ChromaCollector(SentenceTransformerEmbedder()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Programmes\oobabooga_windows\text-generation-webui-main\extensions\superboogav2\chromadb.py", line 101, in __init__ self.collection = self.chroma_client.create_collection(name="context", embedding_function=self.embedder.embed) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Programmes\oobabooga_windows\text-generation-webui-main\installer_files\env\Lib\site-packages\chromadb\api\client.py", line 187, in create_collection return self._server.create_collection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Programmes\oobabooga_windows\text-generation-webui-main\installer_files\env\Lib\site-packages\chromadb\telemetry\opentelemetry\__init__.py", line 127, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "D:\Programmes\oobabooga_windows\text-generation-webui-main\installer_files\env\Lib\site-packages\chromadb\api\segment.py", line 166, in create_collection coll, created = self._sysdb.create_collection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\Programmes\oobabooga_windows\text-generation-webui-main\installer_files\env\Lib\site-packages\chromadb\telemetry\opentelemetry\__init__.py", line 127, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "D:\Programmes\oobabooga_windows\text-generation-webui-main\installer_files\env\Lib\site-packages\chromadb\db\mixins\sysdb.py", line 225, in create_collection raise UniqueConstraintError(f"Collection {name} already exists") chromadb.db.base.UniqueConstraintError: Collection context already exists

I currently can't understand these errors soo i can't get the extention to work (in the example i show superoobaboogav2, but the same error appears with superoobabooga)

[showkeyjar]

chromadb.db.base.UniqueConstraintError: Collection context already exists

same +1

[simaotwx]

looks like im not first, why preinstalled extensions dont have preinstalled requirements whats the point ? Im not a python dev but some devs are forcing it, do i have to add pull request with updated requirements from extensions really ? cmon guys, its not for me, its for non devs cause i already dealt with this If someone wants to fix on its own, activate enviroment, activate.bat is in conda/bin folder F:\oobabooga-windows\installer_files\conda\condabin So, do it like this F:\oobabooga-windows\installer_files\conda\condabin\activate.bat F:\oobabooga-windows\installer_files\env obviousy change the paths to your own, then do - pip install chromadb==0.3.18

This worked like a charm on my Windows 10. Thank you!

Specifically, I ran "cmd_windows.bat", which sets up the environment. I installed the requirements.txt, but ran into the above-mentioned error. After finding this posting, I ran: pip install chromadb==0.3.18

It got further, but still had an error:

2023-10-22 22:43:30 ERROR:Failed to load the extension "superboogav2".
Traceback (most recent call last):
  File "D:\text-generation-webui\modules\extensions.py", line 36, in load_extensions
    exec(f"import extensions.{name}.script")
  File "<string>", line 1, in <module>
  File "D:\text-generation-webui\extensions\superboogav2\script.py", line 21, in <module>
    from .download_urls import feed_url_into_collector
  File "D:\text-generation-webui\extensions\superboogav2\download_urls.py", line 9, in <module>
    from .data_processor import process_and_add_to_collector
  File "D:\text-generation-webui\extensions\superboogav2\data_processor.py", line 12, in <module>
    from .data_preprocessor import TextPreprocessorBuilder, TextSummarizer
  File "D:\text-generation-webui\extensions\superboogav2\data_preprocessor.py", line 14, in <module>
    import pytextrank
ModuleNotFoundError: No module named 'pytextrank'

Ok, fine, I ran: pip install pytextrank Another error, then: pip install optuna

Ok, NOW I think it's happy!

In addition to this, I also had to pip install sentence_transformers

[simaotwx]

Now I'm getting this error:

pydantic.errors.PydanticImportError: `BaseSettings` has been moved to the `pydantic-settings` package. See https://docs.pydantic.dev/2.5/migration/#basesettings-has-moved-to-pydantic-settings for more details.

I appreciate the efforts on developing all of this, however, it is really annoying having to do all of these workarounds. It should just work ootb, especially on Linux where these things tend to be easier to manage.

I had to pip install pydantic==1.10.14 to get this to work. The requirements should be fixed to a certain version to ensure this does not happen.

[moferreira]

I have been trying to run either superbooga or superboogav2 but I can't get it to work. I am using ubuntu 22.04 desktop. I do exactly what is said to be done. Everything works, but superbooga. My basic steps on a fresh ubuntu install are:

sudo apt install python3-pip git curl -y git clone https://github.com/oobabooga/text-generation-webui cd text-generation-webui ./start_linux.sh

Everything gets installed and Ok. Then I stop the running process (CTRL C) and do:

pip install -r extensions/superbooga/requirements.txt

However, the system will always fail to start superbooga(or v2) telling that chromadb is not installed. I have tried to install chromadb in several ways, with no success. The system will always say it is installed but it is more like that I am missing something that is not documented so far.

Can anyone, that got it to work on linux, tell the basic steps to get it working?