citadel-ai / langcheck

Simple, Pythonic building blocks to evaluate LLM applications.
https://langcheck.readthedocs.io/en/latest/index.html
MIT License
184 stars 17 forks source link

Support other types of parameters #152

Closed liwii closed 4 days ago

liwii commented 3 weeks ago

Updated the get_standard_metric_inputs function so that it can any types of inputs flexibly (system prompts, conversation histories, stringified numeric scores... etc) Hopefully with this update LangCheck can help verification of wider variety of LLM-based apps!!

Making use of the infra, I

You can run prompt_leakage by

result_ja = langcheck.metrics.prompt_leakage(
    generated_outputs=generated_outputs,
    system_prompts=system_prompts,
    eval_model=eval_client,
)

You can do the same thing with custom_evaluator by

result_ja = langcheck.metrics.custom_evaluator(
    generated_outputs=generated_outputs,
    prompts=None,
    sources=None,
    reference_outputs=None,
    eval_model=eval_client,
    metric_name="prompt_leakage",
    score_map=score_map,
    template_path="path/to/prompt_leakage.j2",
    language="en",
    additional_params={"system_prompts": system_prompts},
    additional_params_to_prompt_var_mapping={"system_prompts": "system_prompt"},
)