Closed vpicone closed 2 months ago
You can export it for use with llama.cpp with export_gguf
, but I haven't written a method to read that gguf file back in for HuggingFace use yet.
In the meantime, you should be able to use np.save
to save the dataclass as a dictionary, and then reconstitute it that way:
import dataclasses
import numpy as np
...
v = ControlVector.train(...)
np.save("vector.npy", dataclasses.asdict(v))
# later...
v = ControlVector(**np.load("vector.npy", allow_pickle=True).tolist())
Hope this helps!
Hi @vgel,
Thanks for the helpful insight on saving a control vector for later use. I noticed that you mentioned the ability to export a control vector with export_gguf
for use with llama.cpp
. Could you provide an example or guide on how to perform this export?
I found a tutorial on How to convert HuggingFace model to GGUF format. However, I found it difficult to apply this method described here in a Jupyter Notebook (for example, the tutorial ipynb you provided).
Thanks!
@l1mc
Hi @vgel,
Thanks for the helpful insight on saving a control vector for later use. I noticed that you mentioned the ability to export a control vector with
export_gguf
for use withllama.cpp
. Could you provide an example or guide on how to perform this export?I found a tutorial on How to convert HuggingFace model to GGUF format. However, I found it difficult to apply this method described here in a Jupyter Notebook (for example, the tutorial ipynb you provided).
Thanks!
Sorry, didn't see this earlier! Once you have a vector trained with (e.g.) vector = ControlVector.train(...)
, you can simply export it with:
vector.export_gguf("vector.gguf")
If you're running a version with #34 applied, you can also then import the vector back to Python with
vector = ControlVector.import_gguf("vector.gguf")
or you can use it with llama.cpp
using the ./main
runner:
$ ./main ... --control-vector vector.gguf --control-vector-layer-range 14 26 ...
Hope this helps!
Closing this as addressed by #34
Perhaps a naive question, but rather than training a control vector with each run. How might I go about saving it for inference later?