Upgrade Think Thoroughly Agent to have batches of scenarios summing to prob 1

[coderabbitai[bot]]

The Think Thoroughly agent needs an upgrade to ensure that it generates subquestions in batches where the probabilities sum up to 1. Additionally, it is necessary to check the performance of the current version of this agent. Consider deploying the new version as a separate agent, instead of updating the existing one. This issue was requested by @kongzii in PR https://github.com/gnosis/prediction-market-agent/pull/104.

[kongzii]

Something to pay attention here is that if we generate N questions, where the probs should sum up to 1. It doesn't mean they will.

E.g. if the first question is Will GNO hit $500?, agent's probability for it can be 70%. And then if we ask Will GNO stay below $500?, agent's probability can be like 60% or anything he wants...

Maybe it could be some iterative mode, where agent would be asked again and again, while getting previous probabilities in the prompt. And this would continue until he generates reasonable distribution?

[gabrielfior]

@kongzii could you remind me what logic we want for these probabilities? In general, we expect the LLM to output probability $p_1$ for each outcome $o_1$, and since those are independent (they are even executed in parallel), it's expected that they don't sum up to 1. The current approach (see here) is to have the final agent decide, based on the probabilities above (not summing to 1) which final action should be performed by the agent. For this, I expected the final agent to have some freedom around the scenarios_with_probabilities, so that it's not a problem that the values don't add up to 1.

Anyways, more details about the reasoning much appreciated.

[kongzii]

That's correct. That was the original idea, and it seems to work nicely.

This issue is about what Evan mentioned in the PR.

What if instead of generating N independent scenarios and then having N independent probabilities. We ask to generate M scenarios where they should sum up to 1.

But because many interesting possible scenarios won't add up to 1, we would need to generate "batches" of such scenarios and within each batch, probabilities should sum up to 1. (and there can also be 1 batch where they don't)

However, to make them sum up to 1 reasonably, we will need some pre-final-decision iterative process to determine the probabilities within the batch.

That could theoretically provide even better information about the final decision.

Anyway, just an idea.

[gabrielfior]

@kongzii is this issue still relevant? In my head the current state of the think-thoroughly agent already encompasses the logic for this issue, i.e. it generates subquestions. The idea around having probabilities that sum up to 1 is not yet there - if this is indeed required, this issue should still remain open.

[kongzii]

is this issue still relevant?

Yep, it is, I think it will be a nice experiment and then addition to ThinkThoroughly. But probably a low priority.