500 Error Chat Completions

[StreamlinedStartup]

Getting a 500 error when using internal_ip/v1/chat/completions and default env variables. Any help would be greatly appreciated!

Request:

{
    "model": "phi-2",
    "messages": [
        {
            "role": "system",
            "content": "What is the capital of Ohio?"
        }
    ],
    "temperature": 1.31,
    "max_tokens": 8192,
    "top_p": 1.0,
    "n": 1,
    "stream": false
}

Docker Logs:

INFO:     192.168.68.84:62622 - "POST /v1/ HTTP/1.1" 404 Not Found
INFO:     192.168.68.84:62623 - "POST /v1/chat HTTP/1.1" 404 Not Found
INFO:     192.168.68.84:62687 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 116, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 746, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 75, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 70, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 299, in app
    raise e
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 294, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/app.py", line 91, in chat_completions
    return LLM(**c.model_dump()).chat(messages=c.messages)
  File "/app/local_llm/__init__.py", line 344, in chat
    data = self.generate(prompt=prompt)
  File "/app/local_llm/__init__.py", line 310, in generate
    formatted_prompt = format_prompt(
TypeError: 'bool' object is not callable

[dspdavinci]

format_prompt was defined as a function above, simply rename the boolean argument:

def generate(self, prompt, flag_format_prompt: bool = True):
    if flag_format_prompt:
        formatted_prompt = format_prompt(
            prompt=prompt,
            prompt_template=self.prompt_template,
            system_message=self.system_message,
        )
    tokens = get_tokens(formatted_prompt if flag_format_prompt else prompt)
    self.params["n_predict"] = int(self.max_tokens) - tokens
    self.params["n_ctx"] = int(self.max_tokens) - tokens
    llm = Llama(**self.params)
    data = llm(prompt=formatted_prompt if flag_format_prompt else prompt)
    data["model"] = self.model_name
    return data

[Josh-XT]

That is what I get for trying to add some quick functionality! It has been updated to fix this issue. Thank you for reporting. Please update and the issue will be gone.