search

withcatai / node-llama-cpp

Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema on the model output on the generation level
https://node-llama-cpp.withcat.ai
MIT License
894 stars 86 forks source link

feat: max GPU layers param #133

Closed quidmonkey closed 6 months ago

quidmonkey commented 9 months ago

Feature Description

Allow an option for LlamaModel to use all available GPU layers.

The Solution

new LlamaModel({
  gpuLayers: -1,  // use all available gpu layers
});

Considered Alternatives

Another number or symbol other than -1 would also work

Additional Context

Using -1 for GPU layers is standard in Python toolchains

Related Features to This Feature Request

Are you willing to resolve this issue by submitting a Pull Request?

No, I don’t have the time and I’m okay to wait for the community / maintainers to resolve this issue.

github-actions[bot] commented 6 months ago

:tada: This issue has been resolved in version 3.0.0-beta.15 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

github-actions[bot] commented 1 week ago

:tada: This PR is included in version 3.0.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: