Open jlin816 opened 2 months ago
As of now, you need to downgrade your vllm version to 0.3.3
I have the same error. But as far as I can see version 0.3.3 of vllm doesn't have model_loader in vllm.model_executor
.
I also have this error, looks like that function got removed from vllm in this pr when they refactored model loading: https://github.com/vllm-project/vllm/pull/4097
It's defined here: https://github.com/vllm-project/vllm/blob/05434764cd99990035779cf9a4ed86623b528825/vllm/model_executor/model_loader.py#L22-L28.
Looks like some other functions also got removed in the same PR like hf_model_weights_iterator
that are also used to load all the models.
Attempting to downgrade to vllm 0.3.3 causing the installation of the most recent version of sglang to hang indefinitely
I too am seeing this just hang indefinitely if I downgrade. Are there any workarounds?
I too am seeing this just hang indefinitely if I downgrade. Are there any workarounds?
My setup is vllm==0.3.3, transformers==4.38, sglang==0.1.14 My sglang script also hangs
I'm getting the following import error:
It looks like
_set_default_torch_dtype
no longer exists in vllm:https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/model_loader/__init__.py
I've tried both
pip install sglang[all]
and installing from source. My versions aresglang==0.1.14
andvllm==0.4.1
.