Can we Run this improved model? gpt4-x-alpaca-13b-native-4bit-128g

[gsgoldma]

https://github.com/oobabooga/text-generation-webui/discussions/727 can it be converted?

[HanClinto]

I am able to run this with llama.cpp using some hints from that repository's alpaca.sh script. I downloaded the linked model and renamed it ggml-gpt4-x-alpaca-13b-q4_1.bin.

My command: ./main -m models/ggml-gpt4-x-alpaca-13b-q4_1.bin --mlock --color -ins -t 7 -f ./prompts/alpaca.txt -b 256 --top_k 10000 --temp 0.2 --repeat_penalty 1

Note that I had to use the --mlock option on this model because it's a 13b model and would otherwise be slow on my laptop. It makes loading a bit slower, but inference runs reasonably quickly.

The result:

> tell me about alpacas
Alpacas are domesticated animals that belong to the camel family. They are native to the Andes mountain range in South America and are valued for their soft and durable fleece. Alpacas are social animals and communicate with one another through body language and vocalizations. They are also known for their gentle temperament and are sought after for their meat, fiber, and companionship.

> Please write a poem about an alpaca who befriends a turtle.
Upon a sunny day, an alpaca named Nina
Spotted a turtle, small and sweet,
They made eye contact, and Nina's heart did soar,
A connection, pure and unforeseen.

Nina approached the turtle, tail in a wag,
Their friendship blossomed, one that's rare,
A curious alpaca and a patient turtle,
A duo that shared laughter and tears.

Seems to be working fine! :)

[cooperdk]

But the answer is, the model cannot be run with alpaca.cpp - it requires llama.cpp.