bkchang
commented
2 months ago We are seeing the same error for LLM, LLMEngine, and AsyncLLMEngine.
Interestingly, we find wrapping everything in the python script within if __name__ == '__main__': can temporarily bypass the issue.
For test.py being the following
import os
from vllm import LLM, SamplingParams
prompts = ['<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful chatbot who always responds to requests.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is ramen?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n']
sampling_params = SamplingParams(temperature=0.0, max_tokens=512)
llm = LLM(
model="meta-llama/Meta-Llama-3-8B-Instruct",
tensor_parallel_size=2,
disable_log_stats=False,
)
outputs = llm.generate(prompts, sampling_params)Running python test.py gives
(venv-vllm) ben@sakura-h100-3:/nvme0n1/ben/test$ python test.py
/nvme0n1/ben/venvs/venv-vllm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
INFO 06-19 01:40:01 config.py:632] Defaulting to use mp for distributed inference
INFO 06-19 01:40:01 llm_engine.py:161] Initializing an LLM engine (v0.5.0.post1) with config: model='meta-llama/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=meta-llama/Meta-Llama-3-8B-Instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
/nvme0n1/ben/venvs/venv-vllm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
INFO 06-19 01:40:04 config.py:632] Defaulting to use mp for distributed inference
INFO 06-19 01:40:04 llm_engine.py:161] Initializing an LLM engine (v0.5.0.post1) with config: model='meta-llama/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=meta-llama/Meta-Llama-3-8B-Instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "/usr/lib/python3.10/runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/nvme0n1/ben/test/test.py", line 8, in <module>
llm = LLM(
File "/nvme0n1/ben/vllm/vllm/entrypoints/llm.py", line 144, in __init__
self.llm_engine = LLMEngine.from_engine_args(
File "/nvme0n1/ben/vllm/vllm/engine/llm_engine.py", line 371, in from_engine_args
engine = cls(
File "/nvme0n1/ben/vllm/vllm/engine/llm_engine.py", line 223, in __init__
self.model_executor = executor_class(
File "/nvme0n1/ben/vllm/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
super().__init__(*args, **kwargs)
File "/nvme0n1/ben/vllm/vllm/executor/executor_base.py", line 41, in __init__
self._init_executor()
File "/nvme0n1/ben/vllm/vllm/executor/multiproc_gpu_executor.py", line 48, in _init_executor
self.workers = [
File "/nvme0n1/ben/vllm/vllm/executor/multiproc_gpu_executor.py", line 49, in <listcomp>
ProcessWorkerWrapper(
File "/nvme0n1/ben/vllm/vllm/executor/multiproc_worker_utils.py", line 162, in __init__
self.process.start()
File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
ERROR 06-19 01:40:05 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 2334411 died, exit code: 1
INFO 06-19 01:40:05 multiproc_worker_utils.py:123] Killing local vLLM worker processesFor test-wrap.py being the following
import os
from vllm import LLM, SamplingParams
if __name__ == '__main__':
prompts = ['<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful chatbot who always responds to requests.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat is ramen?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n']
sampling_params = SamplingParams(temperature=0.0, max_tokens=512)
llm = LLM(
model="meta-llama/Meta-Llama-3-8B-Instruct",
tensor_parallel_size=2,
disable_log_stats=False,
)
outputs = llm.generate(prompts, sampling_params)Running python test-wrap.py gives
(venv-vllm) ben@sakura-h100-3:/nvme0n1/ben/test$ python test-wrapped.py
/nvme0n1/ben/venvs/venv-vllm/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
warnings.warn(
INFO 06-19 01:43:57 config.py:632] Defaulting to use mp for distributed inference
INFO 06-19 01:43:57 llm_engine.py:161] Initializing an LLM engine (v0.5.0.post1) with config: model='meta-llama/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=meta-llama/Meta-Llama-3-8B-Instruct)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:00 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:00 utils.py:672] Found nccl from library libnccl.so.2
INFO 06-19 01:44:00 utils.py:672] Found nccl from library libnccl.so.2
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:00 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-19 01:44:00 pynccl.py:63] vLLM is using nccl==2.20.5
INFO 06-19 01:44:01 custom_all_reduce_utils.py:196] generating GPU P2P access cache in /home/ben/.config/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 06-19 01:44:06 custom_all_reduce_utils.py:208] reading GPU P2P access cache from /home/ben/.config/vllm/gpu_p2p_access_cache_for_0,1.json
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:06 custom_all_reduce_utils.py:208] reading GPU P2P access cache from /home/ben/.config/vllm/gpu_p2p_access_cache_for_0,1.json
INFO 06-19 01:44:07 weight_utils.py:218] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:07 weight_utils.py:218] Using model weights format ['*.safetensors']
INFO 06-19 01:44:09 model_runner.py:160] Loading model weights took 7.4829 GB
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:10 model_runner.py:160] Loading model weights took 7.4829 GB
INFO 06-19 01:44:11 distributed_gpu_executor.py:56] # GPU blocks: 61915, # CPU blocks: 4096
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:12 model_runner.py:889] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:12 model_runner.py:893] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 06-19 01:44:12 model_runner.py:889] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 06-19 01:44:12 model_runner.py:893] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:16 custom_all_reduce.py:267] Registering 2275 cuda graph addresses
INFO 06-19 01:44:16 custom_all_reduce.py:267] Registering 2275 cuda graph addresses
INFO 06-19 01:44:16 model_runner.py:965] Graph capturing finished in 4 secs.
(VllmWorkerProcess pid=2335588) INFO 06-19 01:44:16 model_runner.py:965] Graph capturing finished in 4 secs.
Processed prompts: 100%|███████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.98s/it, est. speed input: 16.18 toks/s, output: 145.13 toks/s]
[rank0]:[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d 'Note that there is a CUDA IPC Tensor Error and a Leaked Semaphore Objects error at the end, but at least the text generation task can be successfully completed.
This might be related to unprotected usage of python's concurrent.futures library, although the only place I could find vllm using this library is here.
youkaichao
jeejeelee
zixuzixu
yaronr
lixiaochuan2020
Your current environment
🐛 Describe the bug
Description
When setting
tensor_parallel_sizeto a value greater than 1, the program gets stuck and raises aRuntimeErrorrelated to the bootstrapping phase of new processes. This issue does not occur when using version v0.4.3, but persists in versions v0.5.0.post1 and v0.5.0.Steps to Reproduce
Install version v0.5.0.post1 or v0.5.0 of the library.
Run the following Python script with
tensor_parallel_sizeset to 2:Expected Behavior
The program should run without any issues regarding process bootstrapping, similar to how it behaves with version v0.4.3.
Observed Behavior
The program raises the following
RuntimeErrorwhentensor_parallel_sizeis set to 2:Environment
Additional Context
Reverting back to version v0.4.3 resolves the issue. It appears there might be a change in how processes are handled in newer versions that could be causing this error.