search

containers / podman-desktop-extension-ai-lab

Work with LLMs on a local environment using containers
https://podman-desktop.io/extensions/ai-lab
Apache License 2.0
144 stars 23 forks source link

Include Granite code-instruct models in catalog. #1157

Open tombenninger opened 3 weeks ago

tombenninger commented 3 weeks ago

Is your enhancement related to a problem? Please describe

IBM's granite "code-instruct" models are now available in 3b, 8b, 20b, and 34b GGUF formats at https://huggingface.co/models?sort=trending&search=ibm-granite+granite+code-instruct+GGUF.

These models would probably be more useful than the granite "lab" 7b model from before the major market release.

Describe the solution you'd like

Include the 4 code instruct models in the model catalog.

Describe alternatives you've considered

The models can be downloaded and manually imported, but this just adds extra steps for the end user.

Additional context

No response

tombenninger commented 3 weeks ago

I just realized this may be too early. The new llama.cpp to support these models needs to get into the llama-cpp-python library and then that needs to be incorporated into the AI Lab extension in order to run these models, but once that is done, I'd like them added to the catalog. =)

tombenninger commented 3 weeks ago

Oh it looks like llama-cpp-python just got updated, and the newest release (0.2.77) should work: