Look at OFVChallenger - Githubissues

gnosis / prediction-market-agent

GNU Lesser General Public License v3.0

22 stars 4 forks source link

Look at OFVChallenger #457

Open kongzii opened 1 week ago

kongzii commented 1 week ago

After a few days:

resolve some markets manually to check his accuracy
verify it's claiming bonds back
check his losses
???

kongzii commented 6 days ago

resolve some markets manually to check his accuracy

I manually annotated 40 questions in Langfuse:

2 are resolved incorrectly, so we have 95% accuracy
7 are resolved differently from Olas' resolver, from these, 5 are correct
- so Olas' resolver has 87.5% accuracy

Unfortunately, no human challenged these 2 wrong answers. I caught the second one just in time and corrected it by myself, but the first one is now finalized wrongly.

I will check another batch of questions again next week.

kongzii commented 6 days ago

verify it's claiming bonds back check his losses

The agent is getting its xDai back, for example, this transaction https://gnosisscan.io/tx/0x0a8f6a00388be5479600cb2503fb6dabef77c78eb1483c9c3bdda9855a2cb67a.

Just that it loses 0.001 xDai every time it posts the same answer as already posted.