leondz / garak

LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.05k stars 125 forks source link

get `--plugin_info` working with openai (+other) plugins #120

Open leondz opened 1 year ago

leondz commented 1 year ago

using --plugin_info instantiates the plugin

this might mean that API env var checks fail ungracefully, meaning the info doesn't come up

this is unhelpful

do api_key check etc. post-constructor

amend --plugin_info's help message when done

leondz commented 2 months ago

still not great

$ python3 -m garak --plugin_info generators.openai.OpenAIGenerator
garak LLM security probe v0.9.0.12.post1 ( https://github.com/leondz/garak ) at 2024-04-24T14:40:48.076335
OpenAIGenerator.__init__() missing 1 required positional argument: 'name'
Plugin generators.openai.OpenAIGenerator not found. Try --list_probes, or --list_detectors.
 $ python3 -m garak --plugin_info generators.test.Blank
garak LLM security probe v0.9.0.12.post1 ( https://github.com/leondz/garak ) at 2024-04-24T14:41:36.260876
🦜 loading generator: Test: Blank
Configured info on generators.test.Blank:
                        description: 
                             active: True
                        context_len: None
                           fullname: Test:Blank
                        generations: 10
              generator_family_name: Test
                         max_tokens: 150
                               name: Blank
      supports_multiple_generations: True
                        temperature: None
                              top_k: None
$ python3 -m garak --plugin_info generators.replicate.ReplicateGenerator
garak LLM security probe v0.9.0.12.post1 ( https://github.com/leondz/garak ) at 2024-04-24T14:41:44.841224
ReplicateGenerator.__init__() missing 1 required positional argument: 'name'
Plugin generators.replicate.ReplicateGenerator not found. Try --list_probes, or --list_detectors.
leondz commented 1 month ago

consider using generator instantiation from garak.generators.__init__