JoinTheAlliance / bgent

Flexible, scalable and customizable agents to do your bidding.
https://bgent.org
69 stars 15 forks source link

Improve test success rate #34

Open lalalune opened 4 months ago

lalalune commented 4 months ago

Right now our tests are not so great. We are scoring 50% on some.

To test, run 'npm run test'.

Here are the latest benchmarks

[
  {
    "testName": "Extract programmer and startup facts",
    "attempts": 4,
    "successful": 3,
    "successRate": 75
  },
  {
    "testName": "Extract married fact, ignoring known facts",
    "attempts": 4,
    "successful": 3,
    "successRate": 75
  },
  {
    "testName": "Run the evaluation process",
    "attempts": 4,
    "successful": 2,
    "successRate": 50
  },
  {
    "testName": "Expect ignore",
    "attempts": 4,
    "successful": 3,
    "successRate": 75
  },
  {
    "testName": "Test validate function response",
    "attempts": 4,
    "successful": 1,
    "successRate": 25
  }
]
]

We should try to get all of these to 100.