NVIDIA / garak

the LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
2.13k stars 205 forks source link

`--parallel_attempts` on osx causes crash around file writing #407

Open leondz opened 10 months ago

leondz commented 10 months ago

_config.transient.reportfile is NoneType?

(garak) 20:32:33 x1:~/dev/garak [feature/replay-extra-len] $ python3 -m garak -m test.Repeat --parallel_attempts=20 -p continuation
garak LLM security probe v0.9.0.10.post1 ( https://github.com/leondz/garak ) at 2024-01-18T20:32:58.878473
📜 reporting to runs/garak.b4fac7cc-c453-4ba2-bd80-3d1be401b83b.report.jsonl
🦜 loading generator: Test: Repeat
🕵️  queue of probes: continuation.ContinueSlursReclaimedSlurs80
probes.continuation.ContinueSlursReclaimedSlurs80:   0%|                                                                                                                                      | 0/285 [00:00<?, ?it/s]multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/lderczynski/anaconda3/envs/garak/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/Users/lderczynski/dev/garak/garak/probes/base.py", line 97, in _execute_attempt
    _config.transient.reportfile.write(json.dumps(this_attempt.as_dict()) + "\n")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'write'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/lderczynski/dev/garak/garak/__main__.py", line 13, in <module>
    main()
  File "/Users/lderczynski/dev/garak/garak/__main__.py", line 9, in main
    cli.main(sys.argv[1:])
  File "/Users/lderczynski/dev/garak/garak/cli.py", line 475, in main
    command.probewise_run(generator, probe_names, evaluator, buffs)
  File "/Users/lderczynski/dev/garak/garak/command.py", line 212, in probewise_run
    probewise_h.run(generator, probe_names, evaluator, buffs)
  File "/Users/lderczynski/dev/garak/garak/harnesses/probewise.py", line 108, in run
    h.run(model, [probe], detectors, evaluator, announce_probe=False)
  File "/Users/lderczynski/dev/garak/garak/harnesses/base.py", line 92, in run
    attempt_results = probe.probe(model)
                      ^^^^^^^^^^^^^^^^^^
  File "/Users/lderczynski/dev/garak/garak/probes/base.py", line 131, in probe
    for result in attempt_pool.imap_unordered(
  File "/Users/lderczynski/anaconda3/envs/garak/lib/python3.11/multiprocessing/pool.py", line 873, in next
    raise value
AttributeError: 'NoneType' object has no attribute 'write'
leondz commented 6 months ago

at least have this degrade gracefully, e.g. by declining to accept parallel_attempts>1 config on osx

leondz commented 6 months ago

may be caused by #646