not enough swap space issue

[ggbetz]

when evaluating microsoft/orca-7b

[ggbetz]

see also https://github.com/vllm-project/vllm/issues/787

There's an env variable to fix that, so we have to add, e.g.

VLLM_SWAP_SPACE=6

To the configuration (e.g., config.env)

[ggbetz]

The problem seems to arise when traces are generated with beam search. For clarification: https://github.com/vllm-project/vllm/issues/2853

[ggbetz]

This has been resolved.

Pipeline ran successfully for the NousResearch/Nous-Hermes-llama-2-7b model with latest docker container.

Increasing the VLLM_SWAP_SPACE, i.e., the CPU memory vllm may use for offloading during beam search, resolves this issue.

Details:

• On slurm, I had to make sure to grab not only GPUs, but also enough memory allocated to the job via ntasksparameter. Before, I got OOM (not cuda oom) error.
• I used VLLM_SWAP_SPACE=32