search

NumbersStationAI / DuckDB-NSQL

DuckDB NSQL Model
Apache License 2.0
222 stars 13 forks source link

llamafile distribution? #1

Open mskyttner opened 8 months ago

mskyttner commented 8 months ago

I tried these steps and seem to build a llamafile:

cd ~/bin

# download necessary tools and data
curl -L -o llamafile https://github.com/Mozilla-Ocho/llamafile/releases/download/0.6.1/llamafile-0.6.1
curl -L -o zipalign https://github.com/Mozilla-Ocho/llamafile/releases/download/0.6.1/zipalign-0.6.1
curl -L -o ducksql.gguf https://huggingface.co/motherduckdb/DuckDB-NSQL-7B-v0.1-GGUF/resolve/main/DuckDB-NSQL-7B-v0.1-q8_0.gguf?download=true

# verify/test 
chmod +x llamafile zipalign
./llamafile -m ducksql.gguf

# make a single file "ape" executable server
cp llamafile ducksql.llamafile
echo -e "-m\nducksql.gguf\n--host\n0.0.0.0\n..." > .args
zipalign -j0 ducksql.llamafile ducksql.gguf .args

# verify/test
./ducksql.llamafile

It seems to work. Would it make sense to add a GHA to release such a portable llamafile distribution?

Screenshot:

image