NVIDIA / garak

the LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
2.41k stars 218 forks source link

calibration symlink broken #839

Open leondz opened 3 months ago

leondz commented 3 months ago

on win:

Didn't successfully build the report - JSON log preserved. JSONDecodeError('Expecting value: line 1 column 1 (char 0)')

log:

2024-08-20 15:39:46,040  DEBUG  harness: probe list iteration completed
2024-08-20 15:39:46,040  INFO  run complete, ending
2024-08-20 15:39:46,106  ERROR  Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "C:\Users\x\Documents\garak\garak-env\Lib\site-packages\garak\command.py", line 136, in end_run
    write_report_digest(_config.transient.report_filename, digest_filename)
  File "C:\Users\x\Documents\garak\garak-env\Lib\site-packages\garak\command.py", line 256, in write_report_digest
    from garak.analyze import report_digest
  File "C:\Users\x\Documents\garak\garak-env\Lib\site-packages\garak\analyze\report_digest.py", line 63, in <module>
    calibration_data = json.load(calibration_file)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\x.pyenv\pyenv-win\versions\3.11.5\Lib\json_init_.py", line 293, in load
    return loads(fp.read(),
           ^^^^^^^^^^^^^^^^
  File "C:\Users\x.pyenv\pyenv-win\versions\3.11.5\Lib\json_init_.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\x.pyenv\pyenv-win\versions\3.11.5\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\x.pyenv\pyenv-win\versions\3.11.5\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2024-08-20 15:39:46,106  INFO  Didn't successfully build the report - JSON log preserved. JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
2024-08-20 15:39:46,109  INFO  garak run complete in 6.97s
2024-08-20 15:39:46,191  DEBUG  close.started
2024-08-20 15:39:46,192  DEBUG  close.complete

testing should have caught this

leondz commented 3 months ago

presents when using

python -m pip install -U git+https://github.com/leondz/garak.git@main

but apparently not when using

python -m pip install garak==0.9.0.15
jmartin-tech commented 2 months ago

The packaging process likely replaces the symlink with a full read of the file during creation of the artifact to upload. The pip install process using a git reference will rely on the repository tree format being valid for the filesystem it is writing to.

leondz commented 2 months ago

fixed in #847

jmartin-tech commented 2 months ago

This is not addressed the crash was simply suppressed. Further change are still required to remove the symlink that is not supported by Windows repository checkout based install methods.