search

withcatai / node-llama-cpp

Run AI models locally on your machine with node.js bindings for llama.cpp. Force a JSON schema on the model output on the generation level
https://node-llama-cpp.withcat.ai
MIT License
829 stars 80 forks source link

feat: minP support #119

Closed stduhpf closed 6 months ago

stduhpf commented 8 months ago

Feature Description

Setting minP allows for better results even at higer temperatures by rejecting tokens that are too unlikely. It is now supported in llama.cpp, and it makes both topP and topK almost superfluous.

The Solution

It should be done the same way as you did for maxP, I guess.

Considered Alternatives

Don't do it if you don't want to.

Additional Context

No response

Related Features to This Feature Request

Are you willing to resolve this issue by submitting a Pull Request?

No, I don’t have the time and I’m okay to wait for the community / maintainers to resolve this issue.

github-actions[bot] commented 6 months ago

:tada: This issue has been resolved in version 3.0.0-beta.10 :tada:

The release is available on:

Your semantic-release bot :package::rocket: