cublas implemetation?

[yesbroc]

Duplicates

• [X] I have searched the existing issues

Summary 💡

for llama, there's a flag called --gpu-layers N, basically oflloads some layers to the gpu for processing

Examples 🌈

from ooba

Motivation 🔦

since cpu is super slow, gpu would be nice

[rhohndorf]

You have to set some environment variables before llama-cpp-python installation so that it get compiled with cublas support. Follow the instructions here: https://pypi.org/project/llama-cpp-python/