search

su77ungr / CASALIOY

♾️ toolkit for air-gapped LLMs on consumer-grade hardware
Apache License 2.0
229 stars 31 forks source link

Multi-Model Support + .env configuration file #17

Closed alxspiker closed 1 year ago

alxspiker commented 1 year ago

Also added default so if no arguments are supplied to ingest, it just uses the .env DOCUMENTS_DIRECTORY

P.S. Must rename example.env to .env to use

su77ungr commented 1 year ago

thanks for the effort. i was on this right now too haha.

let me run a brief check and then up we go.

alxspiker commented 1 year ago

Got bored working on an auto convert script. I got it done, Ill send a PR if you want to look at it. Basically it uses the .env to look at your LLM model_path and checks the models type. If it is ggml, it converts it to ggjt. If its already ggjt, it does nothing and exits. Could probably call it from the ingest and/or startllm based on a .env CONVERT_OLD_MODEL=True or something. Just an idea.

su77ungr commented 1 year ago

Got bored working on an auto convert script. I got it done, Ill send a PR if you want to look at it. Basically it uses the .env to look at your LLM model_path and checks the models type. If it is ggml, it converts it to ggjt. If its already ggjt, it does nothing and exits. Could probably call it from the ingest and/or startllm based on a .env CONVERT_OLD_MODEL=True or something. Just an idea.

Testing something similiar by probing the binary itself with magicbytes. And auto-detecting if it has a gptj backend.

Glad your are so productive, I'd welcome such an addition. Just set a auto_convert variable inside .env. If true the convertion is done.