search

h2oai / h2ogpt

Private chat with local GPT with document, images, video, etc. 100% private, Apache 2.0. Supports oLLaMa, Mixtral, llama.cpp, and more. Demo: https://gpt.h2o.ai/ https://codellama.h2o.ai/
http://h2o.ai
Apache License 2.0
10.95k stars 1.2k forks source link

Updated version causes error with TGI #1419

Open vitalyshalumov opened 4 months ago

vitalyshalumov commented 4 months ago

I have updated my h2ogpt docker version and now my docker wan't start. The error is thrown : Malformed inference server. It is a bug because few lines before in the command line it succeeded generating text with said inference server.

pseudotensor commented 4 months ago

We use TGI with h2oGPT just fine and we have tests.

That error is only reached if you did not set --inference_server. So must be typo in the CLI options if you are intending to use TGI as inference server.

vitalyshalumov commented 4 months ago

We use TGI with h2oGPT just fine and we have tests.

That error is only reached if you did not set --inference_server. So must be typo in the CLI options if you are intending to use TGI as inference server.

As I mentioned earlier, it worked on previous h2ogpt versions. In addition, it manages to generate a token using TGI at the start of the docker so typo is probably not the issue.

pseudotensor commented 4 months ago

I understand, but the error message is literally only hit when inference_server is not set.

vitalyshalumov commented 4 months ago

Thank you for your time. How does this align with " it manages to generate a token using TGI at the start of the docker"?