search

logikon-ai / cot-eval

A framework for evaluating the effectiveness of chain-of-thought reasoning in language models.
https://huggingface.co/spaces/logikon/open_cot_leaderboard
MIT License
5 stars 1 forks source link

Evaluate: tiiuae/falcon-11B #52

Open ggbetz opened 1 month ago

ggbetz commented 1 month ago

Check upon issue creation:

Parameters:

NEXT_MODEL_PATH=tiiuae/falcon-11B
NEXT_MODEL_REVISION=main
NEXT_MODEL_PRECISION=bfloat16
MAX_LENGTH=2048 
GPU_MEMORY_UTILIZATION=0.8
VLLM_SWAP_SPACE=4

ToDos:

yakazimir commented 4 weeks ago

possible issue with VLLM and transformers: 


2024-06-09T00:56:34.810614509Z Traceback (most recent call last):
2024-06-09T00:56:34.810640753Z   File "/usr/local/bin/cot-eval", line 8, in <module>
2024-06-09T00:56:34.810670576Z     sys.exit(main())
2024-06-09T00:56:34.810676624Z   File "/workspace/cot-eval/src/cot_eval/__main__.py", line 149, in main
2024-06-09T00:56:34.810719213Z     llm = VLLM(
2024-06-09T00:56:34.810737842Z   File "/usr/local/lib/python3.10/dist-packages/langchain_core/load/serializable.py", line 120, in __init__
2024-06-09T00:56:34.810753544Z     super().__init__(**kwargs)
2024-06-09T00:56:34.810762650Z   File "/usr/local/lib/python3.10/dist-packages/pydantic/v1/main.py", line 339, in __init__
2024-06-09T00:56:34.810825823Z     values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
2024-06-09T00:56:34.810833903Z   File "/usr/local/lib/python3.10/dist-packages/pydantic/v1/main.py", line 1102, in validate_model
2024-06-09T00:56:34.810972709Z     values = validator(cls_, values)
2024-06-09T00:56:34.810989335Z   File "/usr/local/lib/python3.10/dist-packages/langchain_community/llms/vllm.py", line 88, in validate_environment
2024-06-09T00:56:34.810996428Z     values["client"] = VLLModel(
2024-06-09T00:56:34.810998644Z   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py", line 112, in __init__
2024-06-09T00:56:34.811023086Z     self.llm_engine = LLMEngine.from_engine_args(
2024-06-09T00:56:34.811028309Z   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 196, in from_engine_args
2024-06-09T00:56:34.811084375Z     engine = cls(
2024-06-09T00:56:34.811087001Z   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 110, in __init__
2024-06-09T00:56:34.811088759Z     self.model_executor = executor_class(model_config, cache_config,
2024-06-09T00:56:34.811091133Z   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/ray_gpu_executor.py", line 62, in __init__
2024-06-09T00:56:34.811126708Z     self._init_workers_ray(placement_group)
2024-06-09T00:56:34.811132405Z   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/ray_gpu_executor.py", line 192, in _init_workers_ray
2024-06-09T00:56:34.811168916Z     self._run_workers(
2024-06-09T00:56:34.811174082Z   File "/usr/local/lib/python3.10/dist-packages/vllm/executor/ray_gpu_executor.py", line 324, in _run_workers
2024-06-09T00:56:34.811209017Z     driver_worker_output = getattr(self.driver_worker,
2024-06-09T00:56:34.811214567Z   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 107, in load_model
2024-06-09T00:56:34.811242529Z     self.model_runner.load_model()
2024-06-09T00:56:34.811247480Z   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 95, in load_model
2024-06-09T00:56:34.811256892Z     self.model = get_model(
2024-06-09T00:56:34.811258664Z   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader.py", line 101, in get_model
2024-06-09T00:56:34.811294265Z     model.load_weights(model_config.model, model_config.download_dir,
2024-06-09T00:56:34.811301734Z   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 425, in load_weights
2024-06-09T00:56:34.811364589Z     param = params_dict[name]
2024-06-09T00:56:34.811377353Z KeyError: 'transformer.h.26.input_layernorm.weight'
2024-06-09T00:56:36.929909774Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44] Error executing method load_model. This might cause deadlock in distributed execution.
2024-06-09T00:56:36.929931343Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44] Traceback (most recent call last):
2024-06-09T00:56:36.929933284Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/ray_utils.py", line 37, in execute_method
2024-06-09T00:56:36.929935669Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44]     return executor(*args, **kwargs)
2024-06-09T00:56:36.929937002Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 107, in load_model
2024-06-09T00:56:36.929938537Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44]     self.model_runner.load_model()
2024-06-09T00:56:36.929939834Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 95, in load_model
2024-06-09T00:56:36.929941274Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44]     self.model = get_model(
2024-06-09T00:56:36.929942571Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader.py", line 101, in get_model
2024-06-09T00:56:36.929944200Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44]     model.load_weights(model_config.model, model_config.download_dir,
2024-06-09T00:56:36.929945603Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 425, in load_weights
2024-06-09T00:56:36.929947310Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44]     param = params_dict[name]
2024-06-09T00:56:36.929948607Z (RayWorkerVllm pid=10701) ERROR 06-09 00:56:34 ray_utils.py:44] KeyError: 'transformer.h.26.input_layernorm.weight'
2024-06-09T00:56:36.929950064Z (RayWorkerVllm pid=10931) INFO 06-09 00:56:03 weight_utils.py:177] Using model weights format ['*.safetensors'] [repeated 2x across cluster]
2024-06-09T00:56:36.929952403Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44] Error executing method load_model. This might cause deadlock in distributed execution. [repeated 2x across cluster]
2024-06-09T00:56:36.929953926Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44] Traceback (most recent call last): [repeated 2x across cluster]
2024-06-09T00:56:36.929955302Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/ray_utils.py", line 37, in execute_method [repeated 2x across cluster]
2024-06-09T00:56:36.929964983Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44]     return executor(*args, **kwargs) [repeated 2x across cluster]
2024-06-09T00:56:36.929966426Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44]   File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 95, in load_model [repeated 4x across cluster]
2024-06-09T00:56:36.929968052Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44]     self.model_runner.load_model() [repeated 2x across cluster]
2024-06-09T00:56:36.929969420Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44]     self.model = get_model( [repeated 2x across cluster]
2024-06-09T00:56:36.929970765Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader.py", line 101, in get_model [repeated 2x across cluster]
2024-06-09T00:56:36.929972269Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44]     model.load_weights(model_config.model, model_config.download_dir, [repeated 2x across cluster]
2024-06-09T00:56:36.929973771Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/falcon.py", line 425, in load_weights [repeated 2x across cluster]
2024-06-09T00:56:36.929975266Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44]     param = params_dict[name] [repeated 2x across cluster]
2024-06-09T00:56:36.929976815Z (RayWorkerVllm pid=10931) ERROR 06-09 00:56:35 ray_utils.py:44] KeyError: 'transformer.h.26.input_layernorm.weight' [repeated 2x across cluster]```