search

logikon-ai / cot-eval

A framework for evaluating the effectiveness of chain-of-thought reasoning in language models.
https://huggingface.co/spaces/logikon/open_cot_leaderboard
MIT License
12 stars 2 forks source link

mistralai/Mistral-Large-Instruct-2407 doesn't load #67

Open ggbetz opened 1 month ago

ggbetz commented 1 month ago

vllm server fails to load model in 30min

last logs:

INFO 10-02 08:07:17 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x14fbd35198e0>, local_subscribe_port=37383, remote_subscribe_port=None)
INFO 10-02 08:07:18 model_runner.py:1014] Starting to load model mistralai/Mistral-Large-Instruct-2407...
(VllmWorkerProcess pid=2089121) INFO 10-02 08:07:18 model_runner.py:1014] Starting to load model mistralai/Mistral-Large-Instruct-2407...
(VllmWorkerProcess pid=2089122) INFO 10-02 08:07:18 model_runner.py:1014] Starting to load model mistralai/Mistral-Large-Instruct-2407...
(VllmWorkerProcess pid=2089123) INFO 10-02 08:07:18 model_runner.py:1014] Starting to load model mistralai/Mistral-Large-Instruct-2407...
(VllmWorkerProcess pid=2089122) INFO 10-02 08:07:20 weight_utils.py:242] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=2089121) INFO 10-02 08:07:20 weight_utils.py:242] Using model weights format ['*.safetensors']
(VllmWorkerProcess pid=2089123) INFO 10-02 08:07:20 weight_utils.py:242] Using model weights format ['*.safetensors']
INFO 10-02 08:07:20 weight_utils.py:242] Using model weights format ['*.safetensors']