search

redhat-et / copilot-ops

copilot-ops is a CLI tool that boosts up any DevOps repo to a ninja level of Artificially Intelligent Ops Repo
Apache License 2.0
34 stars 13 forks source link

AI Consistency Testing #15

Closed RobotSail closed 2 years ago

RobotSail commented 2 years ago

Testing that an AI provides the correct response on a given prompt is difficult.

We should develop some system of unit-testing which applies to a stochastic system where the output is never a guarantee. One idea could be setting the temperature parameter to 0 and seeing how often a given response yields certain properties.

Since we are also paying per request, this also becomes a costly endeavor.

RobotSail commented 2 years ago

This would be solved by fine-tuning a model rather than trying to test for consistency