KM evaluation refactorings

[kbeaugrand]

High level description (Approach, Design)

• Add builders for generator and evaluators
• Use DI to create engin classes
• Update sample code + README

[xuzeyu91]

I tried using evaluation, but in the

        return this._serviceCollection.BuildServiceProvider()
            .GetRequiredService<TestSetGenerator>();

There is a dependency injection error here, and I am unable to directly inject TestSetGenerator. What should I do?

[dluc]

@kbeaugrand FYI

[kbeaugrand]

@xuzeyu91, please take a look at the readme. I'll create later a PR for an official documentation

[xuzeyu91]

@xuzeyu91, please take a look at the readme. I'll create later a PR for an official documentation

I didn't understand how to inject this service, and I made the same mistake using the example of 401 directly

https://github.com/microsoft/kernel-memory/blob/main/examples/401-evaluation/Program.cs

[xuzeyu91]

@kbeaugrand

[kbeaugrand]

@xuzeyu91,

Generator and evaluator are not added into the ServiceCollection by default. If this is needed, in your case you should add it on your own.

To instanciate a testSetGenerator you should follow the sample:

using Microsoft.KernelMemory.Evaluation;

var testSetGenerator = new TestSetGeneratorBuilder(memoryBuilder.Services)
                            .AddEvaluatorKernel(kernel)
                            .Build();

var distribution = new Distribution
{
    Simple = .5f,
    Reasoning = .16f,
    MultiContext = .17f,
    Conditioning = .17f
};

var testSet = testSetGenerator.GenerateTestSetsAsync(index: "default", count: 10, retryCount: 3, distribution: distribution);

await foreach (var test in testSet)
{
    Console.WriteLine(test.Question);
}

Maybe I don't understand correctly your issue. Can you give me more explanation about what your are intending to do?

[xuzeyu91]

This doesn't work. Have you ever tested it?