search

Maximilian-Winter / llama-cpp-agent

The llama-cpp-agent framework is a tool designed for easy interaction with Large Language Models (LLMs). Allowing users to chat with LLM models, execute structured function calls and get structured output. Works also with models not fine-tuned to JSON output and function calls.
Other
472 stars 42 forks source link

Is there any way to let the model think before choosing to proceed with a certain function ? #30

Closed reachsak closed 6 months ago

reachsak commented 6 months ago

I have 2 python functions, def agree() and def disagree() In my prompt I've tried giving an opinion and ask if the model agree or disagree with my statement. I've tried using two DynamicSampleModel, but it gave errors. Would like to know if there is a way to implement this. Thank you,

Maximilian-Winter commented 6 months ago

@reachsak You can pass "add_inner_thoughts = True" to the generate_gbnf_grammar_and_documentation function to add a inner thoughts field, which is called "thoughts_and_reasoning" by default. You can change the name of this field to what ever you want by passing inner_thoughts_field_name to the generate_gbnf_grammar_and_documentation function. If you use the function calling agent, it will do it automatically.

Maximilian-Winter commented 6 months ago

@reachsak Can you show me your code? I can help you make it work.

reachsak commented 6 months ago

Thanks, I think I got it working now. I use your " Manual Function Calling with Python Function Example" example and create two dynamicsmodel for each function (agree and disagree) and put in in the function tool list. I gave them prompt like " Canada is a country in Europe, tell me if you agree or disagree to this statement" The model actually has the inner thought as you mentioned above and activate the function.

Now I'm actually having another question on top of this. Is there any way that this code can works with multimodel LLM ? e.g to let the model have inner thought from images. Now I have the LLaVa model running locally on my machine with llamacpp, but I'm not sure how to implement this. Please let me know if you have any workaround, thought on this. Thank you.

Maximilian-Winter commented 6 months ago

Right now the framework only supports text input, but I can look into adding multi modal support.