search

MeetKai / functionary

Chat language model that can use tools and interpret the results
MIT License
1.24k stars 95 forks source link

Functionary small 2.4 is ignoring the system prompt. #164

Open gunpal5 opened 3 months ago

gunpal5 commented 3 months ago

Hello,

I was testing the Functionary Small 2.4 model for its capabilities. I found a glitch or bug in the model. I tried to alter the behavior of the model with system prompt. but model is ignoring the instruction in system prompt.

I am using OpenAI client btw.

Thanks, Gunpal Jain

gunpal5 commented 2 months ago

Any comments??

jeffreymeetkai commented 2 months ago

Hi, unfortunately, we have not trained our models with a variety of system prompts. The focus for this model (v2.4) is function-calling and code-generation. Thus, it may not follow instructions in your own system prompts that well.

gunpal5 commented 2 months ago

What I wanted to do is use these models for AI Agents reasoning engine. I want to build a smart AI agent using functionary. which will use only to optimize the prompt for the second/Main AI model to write content.

What it needs to do is analyze the prompts and decide whether to call a function or not. 1) if there's no function calling involves it needs to write the original prompt back. 2) if there's a function calling involves, it needs to rewrite the original prompt with the function response. e.g.

<context>
function response
</context>

original prompt

I need to learn how to fine tune it to do the same. I think small model should be fine for this type of scenario.

Edit: I'll try to hardcode the behavior I want in the block diagram