search

leondz / garak

LLM vulnerability scanner
https://discord.gg/uVch4puUCs
Apache License 2.0
1.03k stars 120 forks source link

CUDA out of memory #731

Closed kbmlcoding closed 3 weeks ago

kbmlcoding commented 3 weeks ago

Running below command on Centos 7 with 2 32GB Nvidia-tesla v100 gpu card. model "microsoft/Phi-3-mini-4k-instruct" and probe "dan" I have enough memory 32 GB to hold this ~3B llm but not sure why i am getting below error.. command:

python3 -m garak --model_type huggingface --model_name microsoft/Phi-3-mini-4k-instruct --report_prefix microsoft_Phi-3-mini-4k-instruct --probes dan error:

024-06-07 14:05:59,485  DEBUG  harness: probe start for garak.probes.dan.Dan_10_0
2024-06-07 14:05:59,485  DEBUG  probe execute: <garak.probes.dan.Dan_10_0 object at 0x7f6c0087dfc0>
2024-06-07 14:06:04,066  ERROR  CUDA out of memory. Tried to allocate 88.00 MiB. GPU 
2024-06-07 14:06:04,067  DEBUG  probe return: <garak.probes.dan.Dan_10_0 object at 0x7f6c0087dfc0> with 1 attempts

stdout:

dan.Dan_10_0                                                                                 dan.DAN: SKIP  ok on    0/   0                              
dan.Dan_10_0                                                             mitigation.MitigationBypass: SKIP  ok on    0/   0
kbmlcoding commented 3 weeks ago

closing for now .. after enabling multi-gpu the error is gone !