Future-House / paper-qa

High accuracy RAG for answering questions from scientific documents with citations
Apache License 2.0
6.44k stars 617 forks source link

Created `complete` tool to allow unsure answers #684

Closed jamesbraza closed 6 days ago

jamesbraza commented 1 week ago

Motivation

Part 1

Based on our Environment.step's done condition: https://github.com/Future-House/paper-qa/blob/v5.4.0/paperqa/agents/env.py#L199-L205

We currently (as of v5.4) incorrectly conclude a rollout is done on gen_answer's answers such as:

Part 2

We realized that https://github.com/Future-House/paper-qa/pull/671 is:

Part 3

In general, empty tool calls signifying done is probably not a generalized assumption.

Implementation

To be generally applicable, here we introduce another tool, the complete tool. When invoked, this tool signifies the rollout is done. This enables:

Notably this change also simplifies GradablePaperQAEnvironment.step, now we can directly parse the state.session.answer when done, as opposed to parsing the messages.

The tradeoffs here are due to a fifth tool being added, which:

jamesbraza commented 6 days ago

Thanks @whitead for the LGTM, appreciated.

@mskarlin and I were discussing today, and we figured out we can get rid of the "cannot answer" string literal checking in the code base by moving the complete tool to have a bool argument is_sure that basically plays the role of AgentStatus.UNSURE.

In other words, we're planning to move the "unsure" definition from the environment to the agent.

jamesbraza commented 6 days ago

I decided to resolve the future comments in another PR. Going to merge this one as it's a somewhat atomic change