Open giladgd opened 9 months ago
Hey, I have switched to the beta due to the infamous n_tokens <= n_batch
error, and I saw that it is now possible to automatically detect the correct context size. However, there is a problem with that: I have been trying this out with Mistral's OpenOrca 7b in the Q4_K_M quantized size, and the issue is that the training context is 2^15 (32,768), but the quantized version reduces this context to 2,048. With your code example, this will immediately crash the entire server when using your code, since contextSize: Math.min(4096, model.trainContextSize)
will in this case resolve to contextSize: Math.min(4096, 32768)
and then contextSize: 4096
which is > 2048
.
I know that it's not always possible to detect the correct context length, but it would be great if this would not crash the entire app, and instead, e.g., throw an error.
Is it possible to add a mechanism to not crash the module if the provided context size is different from the training context size?
@nathanlesage I'm pretty sure that the reason your app crashes is that larger context size requires more VRAM, and your machine doesn't have enough VRAM for a context length of 4096 but has enough for 2048. If you try to create a context with a larger size than is supported by the model, it won't crash your app but may cause the model to generate gibberish as it crosses the supported context length size.
Unfortunately, it's not possible to safeguard against this at the moment on node-llama-cpp
's side since llama.cpp
is the one that crashes the process, and node-llama-cpp
is not aware of the available VRAM and memory requirements for creating a context with a specific size.
To mitigate this issue I've created this feature request on llama.cpp
: https://github.com/ggerganov/llama.cpp/issues/4315
After this feature is added on llama.cpp
I'll be able to improve this situation on node-llama-cpp
's side.
If this issue is something you expect to happen frequently in your application lifecycle, you can wrap your code with a worker thread until this is fixed properly.
I thought that at first, but then I tried the same code on a windows computer, also with 16 GB of RAM, and it didn't crash. Then I tried out the most recent llama.cpp "manually" (I.e., pulled and ran main) and it worked even with the larger context sizes. I'm beginning to think that this was a bug in the metal code of llama.cpp -- I'll try out beta.2 that you just released, that should fix the issue hopefully.
And thanks for the tip with the worker, I begin to feel a bit stupid for not realizing this earlier, but I've never worked so closely with native code in node before 🙈
@giladgd hi, I think the embedding Fn, can you follow the interface? EmbeddingsInterface here https://github.com/langchain-ai/langchainjs/blob/5df71ccbc734f41b79b486ae89281c86fbb70768/langchain-core/src/embeddings.ts#L9
I'm missing the LlamaContext.decode() function for the tokens, when streaming the prompt using the chat session onToken-"event":
chatSession.prompt(message, onToken(chunk) { console.debug(context.decode(chunk)); // ? //... } }
Is there a substitution for that ?
@mstankala :
model.detokenize(chunk)
transfered to a new discussion https://github.com/withcatai/node-llama-cpp/discussions/176
@scenaristeur As you can see from the logs, node-llama-cpp
detected that you have Vulkan and used it by default.
It's still not smart enough to only offload as much stuff to the GPU as could be fitted in its VRAM, but I plan to implement that in one of the next few beta versions.
For now, you can either disable the GPU support by passing gpu: false
to getLlama
:
import {getLlama} from "node-llama-cpp";
const llama = await getLlama({
gpu: false
});
Or you can lower the context size to make the context consume much less VRAM, to a level that could be fitted in your GPU's VRAM. To inspect how much VRAM you have, you can run this command:
npx --no node-llama-cpp inspect gpu
@scenaristeur As you can see from the logs,
node-llama-cpp
detected that you have Vulkan and used it by default. It's still not smart enough to only offload as much stuff to the GPU as could be fitted in its VRAM, but I plan to implement that in one of the next few beta versions.For now, you can either disable the GPU support by passing
gpu: false
togetLlama
:import {getLlama} from "node-llama-cpp"; const llama = await getLlama({ gpu: false });
Or you can lower the context size to make the context consume much less VRAM, to a level that could be fitted in your GPU's VRAM. To inspect how much VRAM you have, you can run this command:
npx --no node-llama-cpp inspect gpu
it's a 16 core CPU only, no GPU, i'll try getLlama with GPU false . thxs. Perharps i've istalled some Vulkan tools trying some llm but it's a CPU only
@scenaristeur As you can see from the logs,
node-llama-cpp
detected that you have Vulkan and used it by default. It's still not smart enough to only offload as much stuff to the GPU as could be fitted in its VRAM, but I plan to implement that in one of the next few beta versions.For now, you can either disable the GPU support by passing
gpu: false
togetLlama
:import {getLlama} from "node-llama-cpp"; const llama = await getLlama({ gpu: false });
Or you can lower the context size to make the context consume much less VRAM, to a level that could be fitted in your GPU's VRAM. To inspect how much VRAM you have, you can run this command:
npx --no node-llama-cpp inspect gpu
thxs , Works with "gpu:false", but i've lost conversationHistory, how to deal with conversationHistory in the beta version ? I'm working on a server where there can be multiple sessions, with multiple history, in what format should history be injected to a session ? to which class ?
@scenaristeur Please open a discussion for your questions; I don't want to spam with notifications everyone who watches this PR
@scenaristeur As you can see from the logs,
node-llama-cpp
detected that you have Vulkan and used it by default. It's still not smart enough to only offload as much stuff to the GPU as could be fitted in its VRAM, but I plan to implement that in one of the next few beta versions.For now, you can either disable the GPU support by passing
gpu: false
togetLlama
:import {getLlama} from "node-llama-cpp"; const llama = await getLlama({ gpu: false });
Or you can lower the context size to make the context consume much less VRAM, to a level that could be fitted in your GPU's VRAM. To inspect how much VRAM you have, you can run this command:
npx --no node-llama-cpp inspect gpu
thxs ,
Works with "gpu:false", but i've lost conversationHistory, how to deal with conversationHistory in the beta version ? I'm working on a server where there can be multiple sessions, with multiple history, in what format should history be injected to a session ? to which class ?
If your application is GPL 3.0 compliant, feel free to inspire yourself here as to how that can go: https://github.com/nathanlesage/local-chat
@scenaristeur Please open a discussion for your questions; I don't want to spam with notifications everyone who watches this PR
transfered to https://github.com/withcatai/node-llama-cpp/discussions/176
switching from beta 13 to beta 14 gives me
file:///home/smag/dev/igora/node_modules/node-llama-cpp/dist/evaluator/LlamaModel.js:22
constructor({ modelPath, gpuLayers, vocabOnly, useMmap, useMlock, onLoadProgress, loadSignal }, { _llama }) {
^
TypeError: Cannot destructure property '_llama' of 'undefined' as it is undefined.
at new LlamaModel (file:///home/smag/dev/igora/node_modules/node-llama-cpp/dist/evaluator/LlamaModel.js:22:103)
at new McConnector (file:///home/smag/dev/igora/src/mcConnector/index.js:30:13)
at new Worker (file:///home/smag/dev/igora/src/worker/index.js:14:24)
at file:///home/smag/dev/igora/index.js:28:16
Node.js v20.10.0
I am using beta version and the code below still after asking question (giving prompt) i don't get any output its totally blank can anyone please help as I am stuck here badly
import {fileURLToPath} from "url";
import path from "path";
import {getLlama, LlamaChatSession} from "node-llama-cpp";
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const llama = await getLlama({
gpu: false
});
const model = await llama.loadModel({
modelPath: path.join(__dirname, "models", "phi-2-orange.Q2_K.gguf")
});
const context = await model.createContext({
contextSize: Math.min(4096, model.trainContextSize)
});
const session = new LlamaChatSession({
contextSequence: context.getSequence()
});
const q1 = "Hi there, how are you?";
console.log("User: " + q1);
const a1 = await session.prompt(q1);
console.log("AI: " + a1);
const q2 = "Summerize what you said";
console.log("User: " + q2);
const a2 = await session.prompt(q2);
console.log("AI: " + a2);
@juned-adenwalla You can ask it in Discussion.
I'm now using it, works fine with GPU. You might want to look at #199 about __dirname issue. Or try different model like https://huggingface.co/Qwen/Qwen1.5-32B-Chat-GGUF
I have currently two issues preventing me from updating from beta.13 to beta.17.
ggml-meta.metal
library which, as far as I can see, has been replaced with default.metallib
– do you have an idea what might cause this? I am a bit confused that the library would complain about a missing library that appears to have been consciously replaced with a different target…?It would be great if you could give me some pointers so that I can debug it!
(Also: You've mentioned in the changelog to beta.17 that the lib now supports Llama3, but I can confirm that beta.13 works fine with quantized Llama3-models!)
@nathanlesage Have you seen my response for your message in the feedback discussion? I'd like to resolve your issues before I release version 3 as stable.
For anyone who sees this, please share your feedback on the version 3 beta feedback discussion and not on this PR.
How to use this beta
To install the beta version of
node-llama-cpp
, run this command inside of your project:To get started quickly, generate a new project from a template by running this command:
The interface of
node-llama-cpp
will change multiple times before a new stable version is released, so the documentation of the new version will be updated only a bit before the stable version release. If you'd like to use this beta, visit this PR for updated examples of how to use the latest beta version.How you can help
Like this issue to help make it resolve sooner, or open a PR for it: https://github.com/ggerganov/llama.cpp/issues/4315- Implemented innode-llama-cpp
eventuallyIncluded in this beta
init
command to scaffold a project from a template (#217,3.0.0-beta.21
)3.0.0-beta.2
)3.0.0-beta.23
)3.0.0-beta.3
)3.0.0-beta.1
)3.0.0-beta.2
)3.0.0-beta.15
)3.0.0-beta.15
)3.0.0-beta.1
)3.0.0-beta.1
)3.0.0-beta.6
)getLlama
method (#153,3.0.0-beta.6
)3.0.0-beta.10
)3.0.0-beta.11
)3.0.0-beta.23
)3.0.0-beta.23
)3.0.0-beta.13
)3.0.0-beta.12
)3.0.0-beta.14
)inspect gpu
command (#175,3.0.0-beta.13
)inspect gguf
cli command (#182,3.0.0-beta.15
)TemplateChatWrapper
(#175,3.0.0-beta.13
)3.0.0-beta.16
)3.0.0-beta.16
)3.0.0-beta.16
)3.0.0-beta.17
)3.0.0-beta.39
)3.0.0-beta.18
)3.0.0-beta.21
)3.0.0-beta.23
)n_tokens <= n_batch
error (3.0.0-beta.1
)3.0.0-beta.2
)3.0.0-beta.14
)Planned changes before release
LlamaChatSession
llama.cpp
logscontextSize
andbatchSize
defaults that depend on the current machine hardware (#182)CLI usage
Chat with popular recommended models in your terminal with a single command:
Check what GPU devices are automatically detected by
node-llama-cpp
in your project with this command:Usage example
How to stream a response
How to use function calling
Some models have official support for function calling in
node-llama-cpp
(such as Llama 3.1 Instruct and Llama 3 Instruct), while other models fallback to a generic function calling mechanism that works with many models, but not all of them.How to get embedding for text
How to customize binding settings
How to generate a completion
How to generate an infill
Infill, also known as fill-in-middle, is used to generate a completion for an input that should connect to a given continuation. For example, for a prefix input
123
and suffix input789
, the model is expected to generate456
to make the final text be123456789
.Not every model supports infill, so only those that do can be used for generating an infill.
Using a specific compute layer
node-llama-cpp
detects the available compute layers on the system and uses the best one by default. If the best one fails to load, it'll try the next best option and so on until it manages to load the bindings.To use this logic, just use
getLlama
without specifying the compute layer:To force it to load a specific compute layer, you can use the
gpu
parameter ongetLlama
:To inspect what compute layers are detected in your system, you can run this command:
Using
TemplateChatWrapper
To create a simple chat wrapper to use in a
LlamaChatSession
, you can useTemplateChatWrapper
.Example usage:
{{systemPrompt}}
is optional and is replaced with the first system message (when is does, that system message is not included in the history).{{history}}
is replaced with the chat history. Each message in the chat history is converted using template passed tohistoryTemplate
, and all messages are joined together.{{completion}}
is where the model's response is generated. The text that comes after{{completion}}
is used to determine when the model has finished generating the response, and thus is mandatory.functionCallMessageTemplate
is used to specify the format in which functions can be called by the model and how their results are fed to the model after the function call.Using
JinjaTemplateChatWrapper
You can use an existing Jinja template by using
JinjaTemplateChatWrapper
, but note that not all the functionality of Jinja is supported yet. If you want to create a new chat wrapper from scratch, using this chat wrapper is not recommended, and instead you better inherit from theChatWrapper
class and implement a custom chat wrapper of your own in TypeScriptExample usage:
Custom memory management options
node-llama-cpp
adapt to the current free VRAM state to choose the best defaultgpuLayers
andcontextSize
values that maximize those values values within the available VRAM. It's best to not customizegpuLayers
andcontextSize
in order to utilize this feature, but you can also set agpuLayers
value with your constraints, andnode-llama-cpp
will try to adapt to it.node-llama-cpp
also predicts how much VRAM is needed to load a model or create a context when you pass a specificgpuLayers
orcontextSize
value, and throws an error if it's not enough VRAM in order to make sure the process won't crash if there's not enough VRAM. Those estimations are not always accurate, so if you find that it throws an error when it shouldn't, you can passignoreMemorySafetyChecks
to forcenode-llama-cpp
to ignore those checks. Also, in case those calculations are way too inaccurate, please let us know here, and attach the output ofnpx --no node-llama-cpp inspect measure <model path>
with a link to the model file you used.Token bias
Here is an example of to increase the probability of the word "hello" being generated and prevent the word "day" from being generated:
Prompt preloading
Preloading a prompt while the user is still typing can make the model start generating a response to the final prompt much earlier, as it builds most of the context state needed to generate the response.
Prompt completion
Prompt completion is a feature that allows you to generate a completion for a prompt without actually prompting the model.
The completion is context-aware and is generated based on the prompt and the current context state.
When a completion for a prompt there's no use to preloading a prompt before generating a completion for it, as the completion method will preload the prompt automatically.
Pull-Request Checklist
master
branchnpm run format
to apply eslint formattingnpm run test
passes with this changeFixes #0000