search

vgel / repeng

A library for making RepE control vectors
https://vgel.me/posts/representation-engineering/
MIT License
435 stars 31 forks source link

quantized model ? Llama cpp? #2

Closed thiswillbeyourgithub closed 4 months ago

thiswillbeyourgithub commented 5 months ago

Hi,

Reading your articles made me really curious about trying that but I was wondering of it was possible to use HuggingFace's quantized models or even llamacpp or if that required deep changes.

Thanks!

vgel commented 5 months ago

Working on a llama.cpp implementation!

vgel commented 4 months ago

There's now a PR live on the llama.cpp repo: https://github.com/ggerganov/llama.cpp/pull/5970

vgel commented 4 months ago

That PR is merged, so closing this issue.