search

su77ungr / CASALIOY

♾️ toolkit for air-gapped LLMs on consumer-grade hardware
Apache License 2.0
229 stars 32 forks source link

docker/illegal instruction. #23

Closed pinballelectronica closed 1 year ago

pinballelectronica commented 1 year ago

Probably not worth having a Docker image considering the CPU specs varying. I should have known better but FYI

root@6f8561d4692b:/home/CASALIOY# python3 ingest.py /home/casalioy/ llama.cpp: loading model from models/ggml-model-q4_0.bin Illegal instruction

recompiling llama.cpp in theory should work in the container but I didn't bother and built from source.

Great idea and I'm sure a great app lol

su77ungr commented 1 year ago

I spun up an Azure instance that run the tests for me. Sorry for any inconvenience this may have caused. I'm planning to add a label to the release.

For a robust compiler port you might want to check out this ; just found him 5 minutes ago

pinballelectronica commented 1 year ago

Not complaining :) I haven't been able to get CUBLAS working on any of these even compiling with CUBLAS support (Windows) :( Slow ingestion :)

su77ungr commented 1 year ago

switched to python inside the docker setup. Might do the trick for your problem