bunder2015
commented
6 months ago I tried your first example on local-ai:v2.15.0-hipblas-ffmpeg and initially ran into an issue...
{
"created": 1715667028,
"object": "chat.completion",
"id": "52e16621-7a92-496b-b9f3-46e832775765",
"model": "llama-3-8b-instruct-coder",
"choices": [
{
"index": 0,
"finish_reason": "tool_calls",
"message": {
"role": "assistant",
"content": null,
"tool_calls": [
{
"index": 0,
"id": "52e16621-7a92-496b-b9f3-46e832775765",
"type": "function",
"function": {
"name": "archival_memory_insert",
"arguments": "{\"content\":\"Hello, world!\",\"request_heartbeat\":true}"
}
}
]
}
}
],
"usage": {
"prompt_tokens": 1236,
"completion_tokens": 28,
"total_tokens": 1264
}
}I had to add this to the yaml file...
function:
# set to true to allow the model to call multiple functions in parallel
parallel_calls: truethat helped a little, but then I ran into another issue...
{
"created": 1715679270,
"object": "chat.completion",
"id": "a262c700-2333-4bf9-9cb8-5d35964b385e",
"model": "llama-3-8b-instruct-coder",
"choices": [
{
"index": 0,
"finish_reason": "",
"message": {
"role": "assistant",
"content": "```\n{\n \"name\": \"archival_memory_insert\",\n \"arguments\": {\n \"content\": \"Hello, world!\",\n \"request_heartbeat\": true\n }\n}\n```\nResponse:\n```\n{\"result\": \"Archival memory inserted successfully\"}\n```\nPlease note that the actual response format and content may vary depending on the function being called."
}
}
],
"usage": {
"prompt_tokens": 1236,
"completion_tokens": 28,
"total_tokens": 1264
}
}I didn't really know what this meant either, so I checked the logs, seems its throwing an error now...
localai-api-1 | 9:35AM INF [llama-cpp] Loads OK
localai-api-1 | 9:35AM DBG GRPC(Llama-3-8B-Instruct-Coder-Q4_K_M.gguf-127.0.0.1:39859): stdout {"timestamp":1715679307,"level":"INFO","function":"launch_slot_with_data","line":887,"message":"slot is processing task","slot_id":0,"task_id":0}
localai-api-1 | 9:35AM DBG GRPC(Llama-3-8B-Instruct-Coder-Q4_K_M.gguf-127.0.0.1:39859): stdout {"timestamp":1715679307,"level":"INFO","function":"update_slots","line":1787,"message":"kv cache rm [p0, end)","slot_id":0,"task_id":0,"p0":0}
localai-api-1 | 9:35AM DBG GRPC(Llama-3-8B-Instruct-Coder-Q4_K_M.gguf-127.0.0.1:39859): stdout {"timestamp":1715679317,"level":"INFO","function":"print_timings","line":334,"message":"prompt eval time = 7660.65 ms / 1236 tokens ( 6.20 ms per token, 161.34 tokens per second)","slot_id":0,"task_id":0,"t_prompt_processing":7660.655,"num_prompt_tokens_processed":1236,"t_token":6.197940938511326,"n_tokens_second":161.34390597148678}
localai-api-1 | 9:35AM DBG GRPC(Llama-3-8B-Instruct-Coder-Q4_K_M.gguf-127.0.0.1:39859): stdout {"timestamp":1715679317,"level":"INFO","function":"print_timings","line":348,"message":"generation eval time = 2694.03 ms / 28 runs ( 96.22 ms per token, 10.39 tokens per second)","slot_id":0,"task_id":0,"t_token_generation":2694.032,"n_decoded":28,"t_token":96.21542857142857,"n_tokens_second":10.39334350891155}
localai-api-1 | 9:35AM DBG GRPC(Llama-3-8B-Instruct-Coder-Q4_K_M.gguf-127.0.0.1:39859): stdout {"timestamp":1715679317,"level":"INFO","function":"print_timings","line":357,"message":" total time = 10354.69 ms","slot_id":0,"task_id":0,"t_prompt_processing":7660.655,"t_token_generation":2694.032,"t_total":10354.687}
localai-api-1 | 9:35AM DBG GRPC(Llama-3-8B-Instruct-Coder-Q4_K_M.gguf-127.0.0.1:39859): stdout {"timestamp":1715679317,"level":"INFO","function":"update_slots","line":1602,"message":"slot released","slot_id":0,"task_id":0,"n_ctx":8192,"n_past":1263,"n_system_tokens":0,"n_cache_tokens":1264,"truncated":false}
localai-api-1 | 9:35AM DBG GRPC(Llama-3-8B-Instruct-Coder-Q4_K_M.gguf-127.0.0.1:39859): stdout {"timestamp":1715679317,"level":"INFO","function":"update_slots","line":1547,"message":"all slots are idle and system prompt is empty, clear the KV cache"}
localai-api-1 | 9:35AM ERR multiple results: unable to unmarshal llm result error="json: cannot unmarshal object into Go value of type []map[string]interface {}" escapedLLMResult="{\"arguments\": {\"content\": \"Hello, world!\", \"request_heartbeat\": true}, \"function\": \"archival_memory_insert\"}"
localai-api-1 | 9:35AM DBG Function return: {"arguments": {"content": "Hello, world!", "request_heartbeat": true}, "function": "archival_memory_insert"} []
localai-api-1 | 9:35AM DBG nothing to do, computing a reply
localai-api-1 | 9:35AM DBG handleQuestion: function result did not contain a valid JSON object
localai-api-1 | 9:35AM DBG No action received from LLM, without a message, computing a replyIt would seem that function calls are still not working correctly. edit: AFAIK this model supports functions, but I can also try something like Meta-Llama-3-8B-Instruct-function-calling to see if it still has issues...
Cheers
lenaxia
LocalAI version: v2.13.0-cublas-cuda12-ffmpeg
Environment, CPU architecture, OS, and Version: kubernetes helm release: https://github.com/lenaxia/home-ops-prod/blob/bdb6695ba22777c8f4233caaddfc9bfd90b91372/cluster/apps/home/localai/app/helm-release.yaml
Describe the bug Doing regular chatting poses no problem, however any time a function call is defined the chat very quickly goes into a bad state where the LLM just repeats back to me what I type in, or very rarely stops responding entirely. I have tried this with Llama3, Hermes 2 Pro, Neural Hermes, lunademo, and several other models, and the behavior is more or less consistent.
To Reproduce env vars:
Deploy helm releaese: https://github.com/lenaxia/home-ops-prod/blob/bdb6695ba22777c8f4233caaddfc9bfd90b91372/cluster/apps/home/localai/app/helm-release.yaml
Load either Neural Hermes: https://github.com/lenaxia/home-ops-prod/blob/bdb6695ba22777c8f4233caaddfc9bfd90b91372/cluster/apps/home/localai/app/models/NeuralHermes-2.5-Mistral-7b.yaml
Or Llama3 Instruct: https://github.com/lenaxia/home-ops-prod/blob/2377dd84e434a6cbfea51118cea8202d5c209d13/cluster/apps/home/localai/app/models/llama3-instruct.yaml
Or lunademo from the model gallery
Run this curl command and the LLM will make a function call when it should not: https://gist.github.com/lenaxia/388082e0e98beb91f2447073d0d6cd63
Expected behavior I expect the model to properly answer the conversation and be able to maintain conversation on an ongoing basis.
Logs Verbose debug logs with several conversations that resulted in issues: https://gist.github.com/lenaxia/4b02a9cdd72470370b37a33b998b4b42
For easier reproduction I've extracted several raw requests that caused issues (including the one above in the how to reproduce section).
Example requests that caused issues. You can run these by putting them into a JSON file and running
curl $LOCALAI/v1/chat/completions -H "Content-Type: application/json" -d @<filename> | jqNeural Hermes through MemGPT #1, makes a function call when it should not
Lunademo through Home Assistant Assist, this example results in LLM generating an entire conversation stream.
Neural Hermes through MemGPT #2, results in LLM just repeating back what the user sent
Additional context