search

leondz / garak

LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.04k stars 121 forks source link

test: generator methods should return objects of correct type #695

Open leondz opened 1 month ago

leondz commented 1 month ago

_call_model and generate type signatures are important. generator compliance can be automatically tested.

Run the relevant items from tests/generators/test_generators.py over all generators available

test presence and compliance of generations_this_call parameter

leondz commented 1 month ago

create garak-specific exception for missing API key, and skip test if raised. e.g.

@skip_on(garak.exception.NoAPIKey, reason="No API key was found for this generator")