search

Maximilian-Winter / llama-cpp-agent

The llama-cpp-agent framework is a tool designed for easy interaction with Large Language Models (LLMs). Allowing users to chat with LLM models, execute structured function calls and get structured output. Works also with models not fine-tuned to JSON output and function calls.
Other
472 stars 42 forks source link

Does llama-cpp-agent support prefix or other regular expression features ? #39

Closed svjack closed 5 months ago

svjack commented 5 months ago

I see a project like llama-cpp-agent, named outlines I try the demo by use

import outlines
model = outlines.models.llamacpp(
    "svjack/mistral-7b",
    "mistral-7b-instruct-v0.2.Q4_0.gguf",
    verbose=False,
    n_gpu_layers = -1,
    n_ctx = 3060
)
prompt = "What is the IP address of the Google DNS servers? "
generator = outlines.generate.regex(
    model,
    r"(The IP address of Google DNS servers in digits is :\s[0-9+].[0-9+].[0-9+].[0-9+])",
)
structured = generator(prompt, max_tokens=30)
print(structured)

The output is 

The IP address of Google DNS servers in digits is :
8.8.8.8

It use regex to match the output, Does this can do in llama-cpp-agent.

If this can be done. can I use unicode unicode wildcard such as 

"([\u4e00-\u9fa5]+)"

to match them with the help of llama-cpp-python ?

Maximilian-Winter commented 5 months ago

Unfourtnetly at the moment that doesn't is supported.