bug: Concurrent chat doesnt work on Mac Silicon

gabrielle-ong commented 2 weeks ago

Cortex version

1.0.1-203

Describe the Bug

Mac: Concurrent chats for the same model are queued up rather than parallel

Models tested: tinyllama, llama3.2
I expect to open 2 CLI windows / Postman window and have concurrent chats
Works well if separate models (eg tinyllama chat & llama3.2 chat)

May be related to n_parallel parameter in model.yaml

Windows, Ubuntu: Working as expected

Steps to Reproduce

No response

Screenshots / Logs

No response

What is your OS?

[X] MacOS
[ ] Windows
[ ] Linux

What engine are you running?

[X] cortex.llamacpp (default)
[ ] cortex.tensorrt-llm (Nvidia GPUs)
[ ] cortex.onnx (NPUs, DirectML)

gabrielle-ong commented 2 weeks ago

@vansangpfiev do I need to change anything for this to work? I redownloaded the models, but still is non concurrent on my local com and the VM test-macos-13-1 ie right chat finishes, only then left chat begins

gabrielle-ong commented 1 week ago

works with n_parallel = 2, marking as complete

janhq / cortex.cpp