search

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with a KoboldAI UI
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.41k stars 319 forks source link

Constantly switching to swap on macos #754

Closed SimplyCorbett closed 3 months ago

SimplyCorbett commented 3 months ago

I’ve observed that on macOS that when the AI is replying it’s using active memory and when the AI is no longer replying it moves -all- of the active memory in use by koboldccp into swap. Then it moves it back to active memory on the next reply.

as you can imagine this is quickly going to wear out the internal storage.

using the latest release compiled on macOS with metal support and I am using -noblas and gpulayers.

LostRuins commented 3 months ago

You can keep it in memory with --usemlock