Closed kuraisle closed 1 month ago
Currently the EvaluationFramework class doesn't do anything. We need to make it useful.
EvaluationFramework
The behaviour we need is:
PipelineTest
TestPipeline
_save_evaluations()
EvaluationFramework.__init__()
EvaluationFramework.run_evaluations()
EvaluationFramework._save_evaluations()
Is this the right issue type?
Summary
Currently the
EvaluationFramework
class doesn't do anything. We need to make it useful.The behaviour we need is:
PipelineTest
sPipelineTest
and evaluates the outputTestPipeline
and then the pipeline is dropped for RAM purposes, particularly with LLM pipelines_save_evaluations()
should save the results to an output fileAcceptance Criteria
PipelineTest
added toEvaluationFramework.__init__()
EvaluationFramework.run_evaluations()
runs inputEvaluationFramework.run_evaluations()
evaluates outputEvaluationFramework._save_evaluations()
saves json of the output of running evaluationTasks
Confirm creation