socketsupply / socket

A cross-platform runtime for Web developers to build desktop & mobile apps for any OS using any frontend library.
https://socketsupply.co/guides
Other
1.63k stars 78 forks source link

Support Llama.cpp as a core module #912

Closed heapwolf closed 5 months ago

heapwolf commented 9 months ago

This PR will add JavaScript bindings for llama.cpp

import { Model, Context, Grammar } from 'socket:llm'

// include a model file in your project
const model = new Model('./mistral-7b-openorca.Q4_0.gguf')

// create a grammar
const grammar = new Grammar('...') // supports (json)gbnf

// create a context
const context = new Context({ model, grammar })

// use the context to evaluate the input and produce output
context.eval(tokens, params)
ansarizafar commented 8 months ago

Google has recently released gemma models and https://github.com/google/gemma.cpp.

heapwolf commented 5 months ago

added to #next branch