search

empower-ai / empower-functions

GPT-4 level function calling models for real-world tool using use cases
https://www.empower.dev
210 stars 17 forks source link

GPU not used on MacBook Pro M3 Max #4

Closed ChristianWeyer closed 1 month ago

ChristianWeyer commented 1 month ago

When running python -m empower_functions.server --model ggml-model-f16.gguf --chat_format empower-functions

I see that the GPU is not used.

Do we need an extra argument to run with Metal? llama-cpp-python

ChristianWeyer commented 1 month ago

BTW: I installed with:

CMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64 -DLLAMA_METAL=on" pip install --upgrade --verbose --force-reinstall --no-cache-dir llama-cpp-python

ChristianWeyer commented 1 month ago

Got it:

--n_gpu_layers -1