lastmile-ai / aiconfig

AIConfig is a config-based framework to build generative AI applications.
https://aiconfig.lastmileai.dev
MIT License
934 stars 73 forks source link

Convenience Metric constructor(s) #934

Open jonathanlastmileai opened 8 months ago

jonathanlastmileai commented 8 months ago

It would be nice to be able to create a Metric from a function with less required configuration.

  1. Something like generalizing and wrapping the following code with default values:
async def _correct_fn(datum: str, expected: str) -> bool:
    return json.loads(datum) == json.loads(expected)

def correct_function(expected: str):
    return metrics.Metric(
        evaluation_fn=partial(_correct_fn, expected=expected),
        metric_metadata=common.EvaluationMetricMetadata(
            name="correct_function",
            description="True (pass) if function call is correct",
            best_value=True,
            worst_value=False,
            extra_metadata=dict(expected=expected),
        ),
    )

It must expose the comparison function for the user to give in one way or another. Decorator pattern is preferred.

Ankush-lastmile commented 7 months ago

can you provide more context on why this is needed or an example use case?