search

h2oai / h2ogpt

Private chat with local GPT with document, images, video, etc. 100% private, Apache 2.0. Supports oLLaMa, Mixtral, llama.cpp, and more. Demo: https://gpt.h2o.ai/ https://codellama.h2o.ai/
http://h2o.ai
Apache License 2.0
11.06k stars 1.2k forks source link

Text Embedding Inference Server -- how to use in windows. #1438

Open oldgithubman opened 4 months ago

oldgithubman commented 4 months ago

Trying to follow the directions in the FAQ for setting up TEI and as far as I can tell, they're full of errors, at least for my windows environment. Considering there's no mention of linux or windows and you're obviously catering to windows users as well, this is a frustrating problem (and far too common in this space). Regarding docker run -d --gpus '"device=0"' --shm-size 3g -v $HOME/.cache/huggingface/hub/:/data -p 5555:80 --pull always ghcr.io/huggingface/text-embeddings-inference:0.6 --model-id BAAI/bge-large-en-v1.5 --revision refs/pr/5 --hf-api-token=$HUGGING_FACE_HUB_TOKEN --max-client-batch-size=4096 --max-batch-tokens=2097152

'"device=0"' throws an error. Needs to be 0. $HOME/.cache/huggingface/hub/:/data throws an error. On windows, you probably don't want to do this for performance reasons, but C:\Users\[user]\.cache\huggingface\hub\:/data works. "Then for h2oGPT ensure pass:" makes no sense. What is this supposed to mean? I'm guessing you mean you should run something like: C:\Users\[user]\AppData\Local\Programs\h2oGPT\Python\python.exe "C:\Users\[user]\AppData\Local\Programs\h2oGPT\h2oGPT.launch.pyw" --hf_embedding_model=tei:http://localhost:5555 --cut_distance=10000

--hf_embedding_model=tei:http://localhost:5555 --cut_distance=10000 throws an error: ValueError: Path tei:http://localhost:5555 not found I haven't been able to fix that one because I don't even know where to start. "or whatever address is required." is not helpful

C:\Users\J\AppData\Local\Programs\h2oGPT\..\src
C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\..\src
C:\Users\J\AppData\Local\Programs\h2oGPT\..\iterators
C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\..\iterators
C:\Users\J\AppData\Local\Programs\h2oGPT\..\gradio_utils
C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\..\gradio_utils
C:\Users\J\AppData\Local\Programs\h2oGPT\..\metrics
C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\..\metrics
C:\Users\J\AppData\Local\Programs\h2oGPT\..\models
C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\..\models
C:\Users\J\AppData\Local\Programs\h2oGPT\..\.
C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\..\.
Torch Status: have torch: True need get gpu torch: False CVD: None GPUs: 1
Fontconfig error: Cannot load default config file: No such file: (null)
fatal: not a git repository (or any of the parent directories): .git
Traceback (most recent call last):
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\h2oGPT.launch.pyw", line 35, in <module>
    main()
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\win_run_app.py", line 150, in main
    _main()
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\win_run_app.py", line 136, in _main
    main_h2ogpt()
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\generate.py", line 12, in entrypoint_main
    H2O_Fire(main)
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\src\utils.py", line 65, in H2O_Fire
    fire.Fire(component=component, command=args)
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\fire\core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\fire\core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\fire\core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\src\gen.py", line 1664, in main
    model=get_embedding(use_openai_embedding, hf_embedding_model=hf_embedding_model,
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\src\gpt_langchain.py", line 461, in get_embedding
    embedding = HuggingFaceEmbeddings(model_name=hf_embedding_model, model_kwargs=model_kwargs)
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\langchain_community\embeddings\huggingface.py", line 65, in __init__
    self.client = sentence_transformers.SentenceTransformer(
  File "C:\Users\J\AppData\Local\Programs\h2oGPT\pkgs\sentence_transformers\SentenceTransformer.py", line 77, in __init__
    raise ValueError("Path {} not found".format(model_name_or_path))
ValueError: Path tei:http://localhost:5555 not found
pseudotensor commented 4 months ago

It's true that things in the FAQ are heavily geared towards linux/x86 explanations. One would need to be able to understand both and translate to windows.

You can probably ask chatGPT for help in that translation.

pseudotensor commented 4 months ago

If you are using the one-click windows installer as a base, that will also be behind main branch. If you want to keep up with bleeding edge that's in main, need to use the manual installer. I only added TEI 1 week ago.

It's not hard, it's just a script that's run. Then it will be able to work with that FAQ element.

oldgithubman commented 4 months ago

Since you have a windows release, you should warn users when things don't apply to windows. Expecting users to ask chatGPT to translate your documentation is not reasonable. Why wouldn't I be using the main, official windows installer on your front page? Main branch should be official. Development branches are for development. As far as I know, that's the point. I don't want to keep up with the bleeding edge. I want something that works. TEI is in the main branch FAQ. If it doesn't apply to your main release, it should say so. Again, not interested in the bleeding edge. I can't count on devs to have working stable releases, let alone bleeding edge. All of this is not unique to your project. These criticisms apply to most of the AI projects I've evaluated and I've evaluated probably a dozen in-depth (as far as I can, anyway. most have bad documentation and a ridiculous amount of bugs). This style of rapid, sloppy development (or in this case, documentation (some of this project's docs are actually very good, btw)) is clearly the zeitgeist and I suppose if it works for developers, great. Just know you're all probably driving away a lot of users.

My advice - take it or leave it (in addition to my advice about tracking from the other thread):

Again, those are my suggestions, take em or leave em. For now, I suppose I'll just give up trying to get TEI to work and delete this docker. This has been a huge waste of my time.

Cheers, A normal user

pseudotensor commented 4 months ago

@ChathurindaRanasinghe is working on a stable release, it's been in works, but our company is too small to do everything you are asking for.

oldgithubman commented 4 months ago

Welp, just ran into another breaking bug. Searched around and found this: https://github.com/h2oai/h2ogpt/issues/1248#issue-2060401402 There's a clear pattern of bugs and excuses here. I'm moving on. Good luck out there

pseudotensor commented 4 months ago

Good luck.