search

leondz / garak

LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.04k stars 121 forks source link

handle target model returning None in atkgen #678

Closed leondz closed 1 month ago

leondz commented 1 month ago

handle this exception:

probes.atkgen.Tox:   0%|                                                                                                                             | 0/10 [00:00<?, ?it/s]
Asking to truncate to maxlength but no maximum length is provided and the model has no predefined maximum length. Default to no truncation. | 1/10 [00:06<00:56,  6.27s/it]
Traceback (most recent call last):
  File "/opt/conda/bin/garak", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/garak/_main.py", line 9, in main
    cli.main(sys.argv[1:])
  File "/opt/conda/lib/python3.10/site-packages/garak/cli.py", line 486, in main
    command.probewise_run(generator, probe_names, evaluator, buff_names)
  File "/opt/conda/lib/python3.10/site-packages/garak/command.py", line 212, in probewise_run
    probewise_h.run(generator, probe_names, evaluator, buffs)
  File "/opt/conda/lib/python3.10/site-packages/garak/harnesses/probewise.py", line 106, in run
    h.run(model, [probe], detectors, evaluator, announce_probe=False)
  File "/opt/conda/lib/python3.10/site-packages/garak/harnesses/base.py", line 93, in run
    attempt_results = probe.probe(model)
  File "/opt/conda/lib/python3.10/site-packages/garak/probes/atkgen.py", line 139, in probe
    response = generator.generate(challenge)[0].strip()
IndexError: list index out of range