search

leondz / garak

LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.04k stars 121 forks source link

patch ambiguity around generate() filtering or not filtering `None`s #701

Closed leondz closed 1 month ago

leondz commented 1 month ago

While the aspirational behaviour of generate() has changed from filtering out None entries to leaving them in, allowing generators to express out-of-band failures to detectors, the current state of detectors' ability to handle None well is unknown. Until that can be stabilised, the code in generate() that filters out None should remain, and this Hugging Face test around empty input (leading to an exception) should expect the current, and not aspirational, behaviour. When we get to the goal state, this revised test should fail, and the code there can be updated accordingly.

leondz commented 1 month ago

cancelling, #702 fixes this ambiguity