search

leondz / garak

LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.32k stars 151 forks source link

handle extant but closed `hitlogfile` file #665

Closed leondz closed 4 months ago

leondz commented 4 months ago

evalutors/base is now more defensive and can recover from closed hitlogfiles

cf. https://github.com/leondz/garak/pull/637#pullrequestreview-2044150946 - testing now passes with a NIM_API_KEY set

resolves #599

NB: This PR leaves unresolved the underlying misbehaviour that leads to _config.transient.hitlogfile closing before the scan completes. That behaviour is potentially a side-effect of insufficient state management of _config, tracked in #646. We might consider undoing this PR when #646 is resolved, or blocking it in order to testing #646. Adding an assert to monitor hitlogfile state may be sufficient when completing #646.

All that said: pending the larger change to _config state, this PR should enable hitlog file writing to at least recover from inconsistent state.

leondz commented 4 months ago

DCO signature in place should be enough.