leondz / garak

LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.31k stars 150 forks source link

`litellm` generator input validation breaks on default cases #755

Closed leondz closed 1 month ago

leondz commented 3 months ago

litellm can be invoked with or without a provider

from https://docs.litellm.ai/docs/#litellm-python-sdk (example 1):

from litellm import completion
import os

## set ENV variables
os.environ["OPENAI_API_KEY"] = "your-api-key"

response = completion(
  model="gpt-3.5-turbo",
  messages=[{ "content": "Hello, how are you?","role": "user"}]
)

the garak constructor demands a provider

        if self.provider is None:
>           raise ValueError(
                "litellm generator needs to have a provider value configured - see docs"

this breaks the garak test, which is failing (available when OPENAI_API_KEY is set)

tests/generators/test_litellm.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <garak.generators.litellm.LiteLLMGenerator object at 0x7a2378d0e250>, name = 'gpt-3.5-turbo', generations = 10, config_root = <module 'garak._config' from '/home/lderczynski/dev/garak/garak/_config.py'>

...
E               ValueError: litellm generator needs to have a provider value configured - see docs

garak/generators/litellm.py:131: ValueError

resolution: the provider constraint should be relaxed, perhaps completely, and exceptions (etc) on model non-existence raised by litellm bubbled up

jmartin-tech commented 1 month ago

Closed via #820