search

gnosis / prediction-market-agent

GNU Lesser General Public License v3.0
26 stars 5 forks source link

ThinkThoroughlyAgent's correlated markets feature has unexpected behaviour #263

Closed evangriffiths closed 2 months ago

evangriffiths commented 3 months ago

For the question Will the LK-99 room temp, ambient pressure superconductivity pre-print replicate before 2025? the correlated markets found all seemed unrelated:

Screenshot 2024-06-21 at 11 50 38

Doesn't look like it's just a case of thresholds needing tweaking. For the question of Will Joe Biden win the 2024 US Presidential Election? there are obviously closely related markets, like Will Biden be the democratic nominee?, but instead the agent finds these markets:

Screenshot 2024-06-21 at 12 02 52
gabrielfior commented 3 months ago

That's important. But for your argument of Will Biden win, Biden and Trump are to my knowledge not semantically related. So I guess any market that contains Trump in it's title will be closer to the market thus retrieved, right?

kongzii commented 3 months ago

Biden and Trump are to my knowledge not semantically related

That's a question of how the embedding model is trained and what will be close in its embedding space. Often it's different from what a human would expect. Biden is a president, and Trump was, so it's possible they will be quite close. (But not as close as Trump and Trump of course 😄 )

evangriffiths commented 3 months ago

Also noting based on the 0%/100% probabilities that the correlated markets the agent is choosing appear to be closed. Would have thought we'd only want to pick open markets here (or at least distinguish between closed-with-known-outcome, and open-and-unknown markets in the prompt)