search

leondz / garak

LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.31k stars 150 forks source link

tests: check plugin modules for classes that aren't picked up by plugin enumeration #713

Open leondz opened 4 months ago

leondz commented 4 months ago

Classes that aren't picked up by plugin enumeration, aren't tested by tests that use the same plugin enumeration. That's not good.

We'll need to work out what to do with mixins; maybe add a skiplist in the tests.

jmartin-tech commented 2 months ago

The original intent of this may have been addressed in the plugin processing refactors from #711 and #768.

Further evolution of the plugin cache can add more tests to document edge cases that inject new plugins classes in existing modules as well as tests that around validating entirely new modules will properly be enumerated.