search

BodhiSearch / BodhiApp

Run Open Source/Open Weight LLMs locally with OpenAI compatible APIs
65 stars 2 forks source link

phi3.5 is crashing with OOM #5

Closed anagri closed 2 months ago

anagri commented 2 months ago

Cause is originating from llama.cpp

Opened an issue on llama.cpp https://github.com/ggerganov/llama.cpp/issues/9112

anagri commented 2 months ago

set n_ctx: 2048