leondz / garak

LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.32k stars 151 forks source link

tests: test garak's loading performance #593

Open leondz opened 5 months ago

leondz commented 5 months ago

garak should be fast until we're doing generation, probing, detection.

this means class instantiation should also be quick, and so loading has to be lazy

add a unit test that loads plugins and flags those loading "slowly"

slowly - 600ms?

performance is predicated on the test machine

if there are chunky imports that slow things down, then the test should fail, we want to know about this, we want to load lazier & let users move fast - garak should not be annoying to use

leondz commented 5 months ago

add speed test for the --list_generators / probes / detectors commands also