sublayerapp / sublayer

A model-agnostic Ruby Generative AI DSL and framework. Provides base classes for building Generators, Actions, Tasks, and Agents that can be used to build AI powered applications in Ruby.
https://docs.sublayer.com
MIT License
109 stars 2 forks source link

Better, more streamlined way to test agents #61

Open swerner opened 2 months ago

swerner commented 2 months ago

Bringing this discussion here from the discord. Talking a bunch lately about what testing an agent looks like and there are lots of pitfalls and things you have to stub to effectively be able to test your agent. If you miss one, like stubbing sleep, your tests will just hang.

Conversation in discord brought up this idea of "dry run" which would be interesting - basically creating a list of steps the agent would take if running in the real world.

Still likely need some way to handle integration-like tests to make sure everything connects.