search

spcl / graph-of-thoughts

Official Implementation of "Graph of Thoughts: Solving Elaborate Problems with Large Language Models"
https://arxiv.org/pdf/2308.09687.pdf
Other
1.97k stars 141 forks source link

Question about backtracking and looping over thoughts in GoT #8

Closed FruVirus closed 8 months ago

FruVirus commented 9 months ago

Hi there,

First of all, great work on implementing GoT and the publication of your paper---really appreciate you guys making your source code publicly available!

  1. I have a question regarding how one implements backtracking procedures in the Graph of Operations? Technically, you cannot do this (otherwise it wouldn't be a DAG). Thus, is the idea that you define all possible paths and then have Operations effectively halt their execution for paths that fail to meet some criteria?
  2. Second question is regarding looping over thoughts. How does one refine a single Operation? Or does the paper only mean that one can refine over Thoughts and not Operations?
nblach commented 8 months ago

Hi @FruVirus,

Thanks for your kind words and for taking an interest in our project!

  1. You are correct. At this point in time, the GoO is a DAG (+ self-loops) constructed statically before execution. This means that you would have to define the paths you want to explore before execution and utilise filtering operations to remove paths that are not worth exploring further (generally, with some small exceptions, an operation with no previous thoughts will not perform any work).
  2. I am not sure what you mean with refining an operation. Operations deal with processing thoughts. When we talk about "refinement", we mean improving thoughts, not the actions themselves. Check out the ValidateAndImprove Operation. For each thought of the previous operations, it tries to improve the thought until it is valid (or the number of tries reaches the limit). You could implement a similar operation quite easily for continuous refinements until a certain score is reached.

I hope this clears things up! Let me know if you have more questions.

kripper commented 2 months ago

Is the path generation still static (done before execution)? This sounds very limiting. Any plans to improve?

kripper commented 2 months ago

Here is a dynamic GoT implementation of your code: https://github.com/JayceNing/DGoT/blob/a0c364ad94bd4110c83485cd50f2e4ea7bbba9a5/graph_of_thoughts/operations/operations.py#L482

The repo also links to a paper.

Consider merging.