Future-House / paper-qa

High accuracy RAG for answering questions from scientific documents with citations
Apache License 2.0
6.44k stars 617 forks source link

Removed `"cannot answer"` literals and added `reset` tool #698

Closed jamesbraza closed 2 days ago

jamesbraza commented 4 days ago

There many of places where we depended on the string literal "cannot answer" in the qa prompt, mainly the environment being done (prior to https://github.com/Future-House/paper-qa/pull/684) or the answer being considered unsure.

This environment check of "cannot answer" also has some downsides:

  1. A coupling of environment functionality to a caller-specified qa prompt
    • We don't validate that "cannot answer" is present in the qa prompt
  2. Added statefulness to our environment (checks for a string literal in answer)

So, what is "unsure"? Really it should be:

  1. gen_answer tool call updates the answer
  2. Given the answer, the agent (and not the environment's gen_answer tool) decides if an answer was successful
  3. If not successful, agent keeps trying to get better evidence, until it gives up

To resolve this, we moved the unsure call directly to the complete tool. Now:

The "cannot answer" check was mostly easy to remove, other than the AnswerSettings.wipe_context_on_answer_failure, since we no longer have a way of checking unsure within gen_answer.

Since the agent controls unsureness now, we needed to make a new tool: reset, which basically performs the use case of wipe_context_on_answer_failure.


After this PR, we have: