search

VincyZhang / intel-extension-for-transformers

Extending Hugging Face transformers APIs for Transformer-based models and improve the productivity of inference deployment. With extremely compressed models, the toolkit can greatly improve the inference efficiency on Intel platforms.
Apache License 2.0
0 stars 0 forks source link

RuntimeError: Chatbot instance has not been set #35

Open VincyZhang opened 3 months ago

VincyZhang commented 3 months ago

Hello,

I started a neuralchat server with following config with command neuralchat_server start --config_file ./server/config/neuralchat.yaml

Config

host: 0.0.0.0
port: <port> # confirmed valid port  

model_name_or_path: "Intel/gpt-j-6B-int8-dynamic-inc" # "Intel/neural-chat-7b-v3-1"
# tokenizer_name_or_path: ""
# peft_model_path: ""
device: "cpu"

and server starts up after downloading model from HF but produces following error while calling http://<ip>:<port>/v1/chat/completions

Error traceback

INFO:     192.168.178.24:28352 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 412, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
    await route.handle(scope, receive, send)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
    await self.app(scope, receive, send)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/fastapi/routing.py", line 299, in app
    raise e
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/fastapi/routing.py", line 294, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/intel_extension_for_transformers/neural_chat/server/restful/textchat_api.py", line 483, in create_chat_completion
    error_check_ret = await check_model(request)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/intel_extension_for_transformers/neural_chat/server/restful/textchat_api.py", line 106, in check_model
    if request.model in router.get_chatbot().model_name:
                        ^^^^^^^^^^^^^^^^^^^^
  File "/home/bijayregmi/Projects/intel-extension-for-transformers/intel_extension_for_transformers/neural_chat/neuralchat_venv_311/lib/python3.11/site-packages/intel_extension_for_transformers/neural_chat/server/restful/textchat_api.py", line 386, in get_chatbot
    raise RuntimeError("Chatbot instance has not been set.")
RuntimeError: Chatbot instance has not been set.

Here is the example cURL:

cURL call

 curl http://<ip>:<port>/v1/chat/completions     -H "Content-Type: application/json"     -d '{
    "model": "Intel/gpt-j-6B-int8-dynamic-inc",
    "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me about Intel Xeon Scalable Processors."}
    ]
    }'
VincyZhang commented 3 months ago

@NeuralChatBot

NeuralChatBot commented 3 months ago

The error message "Chatbot instance has not been set." suggests that the chatbot instance is not being properly initialized. This could be due to a variety of reasons, such as the model not being loaded correctly, the chatbot not being properly configured, or the server not being properly started.

Here are some steps you can take to troubleshoot this issue:

  1. Check the model path: Ensure that the model path specified in the configuration file is correct. You can try downloading the model manually and specifying the local path in the configuration file.

  2. Check the server configuration: Make sure that the server is properly configured to use the specified model. This includes checking the configuration file and ensuring that the model name or path is correctly specified.

  3. Check the server startup: Ensure that the server is starting without errors. You can check the server logs for any error messages that might give clues as to what is going wrong.

  4. Check the chatbot instance: The error message "Chatbot instance has not been set." suggests that the chatbot instance is not being properly initialized. This could be due to a variety of reasons, such as the model not being loaded correctly, the chatbot not being properly configured, or the server not being properly started. You can try to initialize the chatbot instance manually in your code and see if that resolves the issue.

  5. Check the server code: The error might be in the server code. You can try to debug the server code to see if it is correctly initializing the chatbot instance. You can also try to reproduce the issue in a simpler environment to isolate the problem.

If you are still unable to resolve the issue, you might want to consider reaching out to the maintainers of the neuralchat project for further help. They might be able to provide more specific guidance based on the details of your setup and the error message.