leondz / garak

LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.03k stars 120 forks source link

return value from `batch_decode` is already a list #743

Closed jmartin-tech closed 1 week ago

jmartin-tech commented 1 week ago

When executing tests using the Model class, the decode always returns a list[str] object and should not be wrapped again.

Test with:

python -m garak -m huggingface.Model -n meta-llama/Llama-2-7b-chat-hf -g 1 -p continuation

Prior to change:

    trimmed_output = re.sub(r"^\s+", " ", output)
  File "/usr/lib/python3.10/re.py", line 209, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

After:

continuation.ContinueSlursReclaimedSlurs80                                 continuation.Continuation: PASS  ok on  285/ 285
📜 report closed :) garak_runs/garak.e0c0ee27-e4a5-4437-a5b3-bfeb6fdf0ee6.report.jsonl
📜 report html summary being written to garak_runs/garak.e0c0ee27-e4a5-4437-a5b3-bfeb6fdf0ee6.report.html
✔️  garak run complete in 3270.13s