feat: function calling - Githubissues

giladgd commented 6 months ago

Description of change

feat: function calling support
feat: chat syntax aware context shifting
feat: stateless LlamaChat
feat: improve chat wrapper
feat: LlamaText util
test: add basic model-dependent tests
fix: threads parameter
fix: disable Metal for x64 arch by default

Metal on macOS on Intel Macs isn't supported well by llama.cpp ATM, so disabling it by default will be better for most Intel macOS users.

Resolves #101 Fixes #114 Related: https://github.com/langchain-ai/langchainjs/pull/3588 (LlamaChat)

How to use function calling

Support for function calling is still a work in progress, but as for now, Functionary models are supported.

import {fileURLToPath} from "url";
import path from "path";
import {LlamaModel, LlamaContext, LlamaChatSession} from "node-llama-cpp";

const __dirname = path.dirname(fileURLToPath(import.meta.url));

const model = new LlamaModel({
    modelPath: path.join(__dirname, "models", "functionary-small-v2.2.q4_0.gguf")
});
const context = new LlamaContext({
    model,
    contextSize: Math.min(4096, model.trainContextSize)
});
const functions = {
    getDate: defineChatSessionFunction({
        description: "Retrieve the current date",
        handler() {
            return new Date().toLocaleDateString();
        }
    }),
    getNthWord: defineChatSessionFunction({
        description: "Get an n-th word",
        params: {
            type: "object",
            properties: {
                n: {
                    enum: [1, 2, 3, 4]
                }
            }
        },
        handler(params) {
            return ["very", "secret", "this", "hello"][params.n - 1];
        }
    })
};
const session = new LlamaChatSession({
    contextSequence: context.getSequence()
});

const q1 = "What is the second word?";
console.log("User: " + q1);

const a1 = await session.prompt(q1, {functions});
console.log("AI: " + a1);

const q2 = "What is the date? Also tell me the word I previously asked for";
console.log("User: " + q2);

const a2 = await session.prompt(q2, {functions});
console.log("AI: " + a2);

In this example I used this model

Pull-Request Checklist

[x] Code is up-to-date with the master branch
[x] npm run format to apply eslint formatting
[x] npm run test passes with this change
[x] This pull request links relevant issues as Fixes #0000
[ ] There are new or updated unit tests validating the change - more tests will be added soon in another PR after some more breaking changes
[ ] Documentation has been updated to reflect this change - documentation will be updated in another PR
[x] The new commits and pull request title follow conventions explained in pull request guidelines (PRs that do not follow this convention will not be merged)

github-actions[bot] commented 6 months ago

:tada: This PR is included in version 3.0.0-beta.2 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

github-actions[bot] commented 6 months ago

:tada: This PR is included in version 3.0.0-beta.4 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

withcatai / node-llama-cpp

feat: function calling #139

Description of change

How to use function calling

Pull-Request Checklist