Open sleepwalker2017 opened 6 months ago
what's the version of nsys? I had a similar error with 2024.1.1, but it worked fine with 2023.4.1
what's the version of nsys? I had a similar error with 2024.1.1, but it worked fine with 2023.4.1
NVIDIA Nsight Systems version 2023.3.1.92-233133147223v0
You only try another version nsys? no other modifications?
In my case, nsys threw an error when TP>1. It was solved by just changing the nsys version without any other modifications.
In my case, nsys threw an error when TP>1. It was solved by just changing the nsys version without any other modifications.
Hi, does it throw error when you using the latest version with tp=1?
Tested on v0.3.2, not the latest version of main branch.
$ nsys --version
NVIDIA Nsight Systems version 2023.4.1.97-234133557503v0
$ python -c 'import vllm; print(vllm.__version__)'
0.3.2
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Mon_Apr__3_17:16:06_PDT_2023
Cuda compilation tools, release 12.1, V12.1.105
Build cuda_12.1.r12.1/compiler.32688072_0
from vllm import LLMEngine, EngineArgs
def initialize_engine() -> LLMEngine:
engine_args = EngineArgs(model="lmsys/vicuna-7b-v1.5",
max_num_seqs=256,
enforce_eager=True
)
return LLMEngine.from_engine_args(engine_args)
initialize_engine()
$ nsys nvprof python test.py
WARNING: python and any of its children processes will be profiled.
INFO 03-11 20:36:23 llm_engine.py:79] Initializing an LLM engine with config: model='lmsys/vicuna-7b-v1.5', tokenizer='lmsys/vicuna-7b-v1.5', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, seed=0)
INFO 03-11 20:36:30 weight_utils.py:163] Using model weights format ['*.bin']
INFO 03-11 20:36:39 llm_engine.py:337] # GPU blocks: 7381, # CPU blocks: 512
Generating '/tmp/nsys-report-9d38.qdstrm'
[1/7] [========================100%] report1.nsys-rep
[2/7] [========================100%] report1.sqlite
[3/7] Executing 'nvtx_sum' stats report
Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Style Range
-------- --------------- --------- ----------- ----------- --------- --------- ----------- ------- ---------------------------
63.5 986343987 1 986343987.0 986343987.0 986343987 986343987 0.0 PushPop NCCL:ncclCommInitRankConfig
32.8 509790568 1 509790568.0 509790568.0 509790568 509790568 0.0 PushPop NCCL:ncclCommAbort
3.7 57223271 2 28611635.5 28611635.5 902 57222369 40461687.3 PushPop NCCL:ncclGroupEnd
0.0 6502 2 3251.0 3251.0 2074 4428 1664.5 PushPop NCCL:ncclGroupStart
0.0 5621 1 5621.0 5621.0 5621 5621 0.0 PushPop NCCL:ncclAllReduce
... <truncated>
Tested on v0.3.2, not the latest version of main branch.
$ nsys --version NVIDIA Nsight Systems version 2023.4.1.97-234133557503v0 $ python -c 'import vllm; print(vllm.__version__)' 0.3.2 $ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2023 NVIDIA Corporation Built on Mon_Apr__3_17:16:06_PDT_2023 Cuda compilation tools, release 12.1, V12.1.105 Build cuda_12.1.r12.1/compiler.32688072_0
from vllm import LLMEngine, EngineArgs def initialize_engine() -> LLMEngine: engine_args = EngineArgs(model="lmsys/vicuna-7b-v1.5", max_num_seqs=256, enforce_eager=True ) return LLMEngine.from_engine_args(engine_args) initialize_engine()
$ nsys nvprof python test.py WARNING: python and any of its children processes will be profiled. INFO 03-11 20:36:23 llm_engine.py:79] Initializing an LLM engine with config: model='lmsys/vicuna-7b-v1.5', tokenizer='lmsys/vicuna-7b-v1.5', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cuda, seed=0) INFO 03-11 20:36:30 weight_utils.py:163] Using model weights format ['*.bin'] INFO 03-11 20:36:39 llm_engine.py:337] # GPU blocks: 7381, # CPU blocks: 512 Generating '/tmp/nsys-report-9d38.qdstrm' [1/7] [========================100%] report1.nsys-rep [2/7] [========================100%] report1.sqlite [3/7] Executing 'nvtx_sum' stats report Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Style Range -------- --------------- --------- ----------- ----------- --------- --------- ----------- ------- --------------------------- 63.5 986343987 1 986343987.0 986343987.0 986343987 986343987 0.0 PushPop NCCL:ncclCommInitRankConfig 32.8 509790568 1 509790568.0 509790568.0 509790568 509790568 0.0 PushPop NCCL:ncclCommAbort 3.7 57223271 2 28611635.5 28611635.5 902 57222369 40461687.3 PushPop NCCL:ncclGroupEnd 0.0 6502 2 3251.0 3251.0 2074 4428 1664.5 PushPop NCCL:ncclGroupStart 0.0 5621 1 5621.0 5621.0 5621 5621 0.0 PushPop NCCL:ncclAllReduce ... <truncated>
Thank you! I tried with tp=1, nsys profile could run well in this case.
I want to use nsight system to profile vllm.
Here is my code: Seems by default, Ray is not used, is that right? But my nsys profile still fails. Why is that?
Thank you.