search

laelhalawani / glai

glai - GGUF LLAMA AI - Package for simplified model handling and text generation with Llama models quantized to GGUF format. APIs for downloading and loading models automatically, includes a db with models of various scale and quantizations. With this high level API you need one line to load the model and one to generate text completions.
https://laelhalawani.github.io/glai/
Other
5 stars 0 forks source link

Model DB - change the default model db dir to package dir and improve dir location logic #29

Closed laelhalawani closed 6 months ago

laelhalawani commented 6 months ago

Currently the default value of model_db_dir:str is DEFAULT_LOCAL_GGUF_DIR = join_paths("./", "gguf_db"), so the path to the script directory of whichever script is calling the initialization of the ModelDB. This was initially intended to expose the models and provide per project set of models and configs for them, however in testing this doesn't seem that helpful and I think it will be more reasonable to download gguf models to the shared package directory, from which any project using ModelDB can access it. This change requires as little as changing the default value of an argument, the implementation is already compatible with this setup. Here's how it works now if you pass it the directory of the package db 

from glai import EasyAI, ModelDB
from glai.back_end.model_db.db_settings import DEFAULT_LOCAL_GGUF_DIR, MODEL_EXAMPLES_DB_DIR

mdb = ModelDB(model_db_dir=MODEL_EXAMPLES_DB_DIR)

Console

ModelDB dir set to C:\Users\<username>\Anaconda3\envs\<env_name>\Lib\site-packages\glai\back_end\model_db\gguf_models.
Using default model db dir: C:\Users\<username>\Anaconda3\envs\<env_name>\Lib\site-packages\glai\back_end\model_db\gguf_models, changes here will be visible and accessible globally. If you would like to work with a specific db_dir provide it as an argument to the ModelDB constructor.
Loaded 13 models from C:\Users\<username>\Anaconda3\envs\<env_name>\Lib\site-packages\glai\back_end\model_db\gguf_models.

And by passing a different value during initialization it will still be possible to create a db dir with copy of the entries from the package db in the current project as described by the message in the log. However this logic should be extended to also copy the ggufs (based on some bool argument), or if db dir is custom then try first to copy from package db then download if not found there.

laelhalawani commented 6 months ago

The defualt model db dir was change to packages dir, so now it's global for and any project importing AutoAI, EasyAI or ModelDB can access these models. As mentioend is still possible to init with a different dir and use that for models db. That's when the model data jsons would be copied/created and ggufs downloaded.