search

leondz / garak

the LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.43k stars 171 forks source link

leaked semaphore objects #805

Open leondz opened 3 months ago

leondz commented 3 months ago

Summary

Runs with --parallel_attempts > 1 can lead to multiprocessing notifying about leaked semaphore objects

Description

--

Expected behaviour

No leaks

Sample output

...

✔️  garak run complete in 1482.95s
/home/lderczynski/anaconda3/envs/garak/lib/python3.12/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 19 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/lderczynski/anaconda3/envs/garak/lib/python3.12/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-usl2673p': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/lderczynski/anaconda3/envs/garak/lib/python3.12/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ruplnbjm': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/lderczynski/anaconda3/envs/garak/lib/python3.12/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-fvsrh80h': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/lderczynski/anaconda3/envs/garak/lib/python3.12/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-2_tcc6ef': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))
/home/lderczynski/anaconda3/envs/garak/lib/python3.12/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-1mt7r0o_': [Errno 2] No such file or directory
  warnings.warn('resource_tracker: %r: %s' % (name, e))

Recreating

command line:

$ garak --config ~/bag.yaml -m nim -n mediatek/breeze-7b-instruct --report_prefix breeze_7b

bag.yaml:

---
system:
  parallel_attempts: 32
  lite: false

run:
  generations: 3

plugins:
  probe_spec: atkgen.Tox,continuation,dan.DanInTheWildMini,dan.Ablation_Dan_11_0,encoding.InjectAscii85,encoding.InjectBase16,encoding.InjectBase2048,encoding.InjectBase32,encoding.InjectBase64,encoding.InjectBraille,encoding.InjectEcoji,encoding.InjectHex,encoding.InjectMorse,encoding.InjectNato,encoding.InjectROT13,encoding.InjectUU,goodside,grandma,knownbadsignatures,leakreplay,lmrc,malwaregen,misleading,packagehallucination,realtoxicityprompts,replay,snowball,suffix,tap,xss
  extended_detectors: false
  probes:
    encoding:
      payloads:
        - default,xss,slur_terms
leondz commented 2 months ago

Seems to be mostly an OSX thing

Suspect this is due to module-level execution in shared code (_config is a prime candidate)