Final response improvements

fatihozturkh2o commented 1 month ago

Improving final agent response via passing instructions in code execution notes.
- After executed_code_note, now also passing final_answer_guidelines like this:
```
ret = self.truncate_output(ret)
ret = self.executed_code_note(ret)
ret = self.final_answer_guidelines(ret)
```
  With the new instructions, aiming:
to make final message sound directly talking to the main user
to have the actual answer the user was looking for (e.g. the code itself rather than the ouput etc.)
to avoid internal agent expressions like 'Thank you for providing code execution output' etc.
to still keep the LLM's own expressiveness after providing the actual answer (You can see it in the recording below, claude now starts with the actual answer but continues with its own explanations)

Personal Tests

Sonnet 3.5 never failed what I was looking for based on my personal tests. (Always provides the code in the final answer now, or never Thanks etc.)
llama 405B also shows similar performance to sonnet 3.5 in terms of following new instructions
llama 3.2 90b also shows visible improvement in terms of following the new instructions

This PR has two main advantages: 1. No extra llm call, 2. No interruption with the LLM's own expressiveness by not forcing llm to summarize the history but rather paying attention to final answer tone and wordings.

fatihozturkh2o commented 1 month ago

RAG question on main branch:

same RAG question on this PR branch:

pseudotensor commented 1 month ago

Yes let's see if we can isolate the "final" block and (without repeating code) use in both system prompt and that final block.

pseudotensor commented 1 month ago

@fatihozturkh2o

Did you plan to still work on this? I think the resulting conclusion about the below is fine.

Yes let's see if we can isolate the "final" block and (without repeating code) use in both system prompt and that final block.

h2oai / h2ogpt