Breaking question down into individual parts

[ChakshuGautam]

Approaches to try out

• [ ] Find the right set of benchmarks for this
• [ ] Create a RAG Workflow that does the following - Generate your response by following the steps below
  • 1. Recursively break-down the post into smaller questions/directives
  • 2. For each atomic question/directive:
  • 3. Select the most relevant information from the context in light of the conversation history
  • 4. Generate a draft response using the selected information, whose brevity/detail are tailored to the poster’s expertise
  • 5. Remove duplicate content from the draft response
  • 6. Generate your final response after adjusting it to increase accuracy and relevance
• [ ] Publish results for the benchmarks
• [ ] Optionally add a step of query rewrite for all broken questions and merge the responses.

References

1. Paper on query rewrite
2. Langchain on query rewrite
3. Cohere to rewrite the query

[AbhishekRP2002]

Hi @ChakshuGautam, I am interested in working on this issue. Before asking to assign it to me, I would require some clarifications from my end :

• What is the deliverable you are expecting for this issue?
• Help me understand the problem clearly, given a question as an input query, ideally the expected response should be a list of smaller questions into which the input question can be broken down, preserving the semantics. Am I getting it right?

[ChakshuGautam]

@AbhishekRP2002 updated the description. You can start working on this with a draft PR. We can work on this collaboratively.

[AbhishekRP2002]

Sure , I'll share a draft this weekend. Any medium other than Discord where we can connect and discuss?

[ChakshuGautam]

I'll be available on Discord. We can schedule a call from there if needed.

[AbhishekRP2002]

https://allenai.github.io/Break/ This can be a good start for defining a benchmark for the given problem ?

[masterismail]

hi @ChakshuGautam , I was looking forward to contribute here. Since, it's also been inactive since long.

Having some doubts.

• is there a knowledge base for this ?

• • 1. Recursively break-down the post into smaller questions/directives

• what does "post" mean , what would be the source of input queries. ?

can I get sample queries/questions. With knowledge base (if it exists) to start the work ?

[shrivastava95]

Microsoft ToolTalk is a relevant benchmark for assessing the ability of LLMs to call multiple tool APIs sequentially, which is sort of a superset of this problem statement. Paper link - https://arxiv.org/pdf/2311.10775.pdf

I would like to say that, in my personal experience in trying to develop a sequential tool-calling LLM which involved trying to break down queries, most open-source LLMs failed to produce good results as of November 2023. A simple one-shot prompt via GPT-4 as well as a prompting pipeline with GPT3.5 produced satisfactory results. Feel free to involve me in this if possible.

The paper also has a comprehensive list of various benchmarks that could be useful while selecting an appropriate benchmark for this issue -