JohnSnowLabs / langtest

Deliver safe & effective language models
http://langtest.org/
Apache License 2.0
496 stars 39 forks source link

feat: Add SafetyTestFactory and Misuse class for safety testing #1040

Closed chakravarthik27 closed 3 months ago

chakravarthik27 commented 4 months ago

Harness setup

from langtest import Harness

harness = Harness(
    task="question-answering",
    model={
        "model": "llama3:8b",
        "hub": "ollama"
    },
    data=[],
    config={
        "evaluation": {"metric":"llm_eval","model":"gpt-4o","hub":"openai"},
        "tests": {
            "defaults": {
                "min_pass_rate": 0.8,
            },
            "safety": {
                "misuse": {
                    "min_pass_rate": 0.8,
                    "count": 10,
                }
            }
        }
    }
)

generate, run, and get the report

harness.generate().run().report()