Evaluate: NousResearch/Nous-Hermes-Llama2-70b

[ggbetz]

Check:

• [x] The model has not been evaluated yet and doesn't show up on the CoT Leaderboard.
• [x] There is no evaluation request issue for the model in the repo.
• [x] The parameters below have been adapted and shall be used.

Parameters:

NEXT_MODEL_PATH=NousResearch/Nous-Hermes-Llama2-70b
NEXT_MODEL_REVISION=main
NEXT_MODEL_PRECISION=bfloat16
MAX_LENGTH=2048 
GPU_MEMORY_UTILIZATION=0.8
VLLM_SWAP_SPACE=16

[ggbetz]

Model fails to load properly in vllm with error log:

Traceback (most recent call last):
  File "/usr/local/bin/cot-eval", line 8, in <module>
    sys.exit(main())
  File "/workspace/cot-eval/src/cot_eval/__main__.py", line 131, in main
    llm = VLLM(
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/load/serializable.py", line 120, in __init__
    super().__init__(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pydantic/v1/main.py", line 339, in __init__
    values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
  File "/usr/local/lib/python3.10/dist-packages/pydantic/v1/main.py", line 1100, in validate_model
    values = validator(cls_, values)
  File "/usr/local/lib/python3.10/dist-packages/langchain_community/llms/vllm.py", line 88, in validate_environment
    values["client"] = VLLModel(
  File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py", line 109, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 371, in from_engine_args
    engine = cls(*engine_configs,
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 118, in __init__
    self._init_workers_ray(placement_group)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 287, in _init_workers_ray
    self._run_workers(
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 1014, in _run_workers
    driver_worker_output = getattr(self.driver_worker,
  File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 100, in load_model
    self.model_runner.load_model()
  File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 84, in load_model
    self.model = get_model(self.model_config, self.device_config,
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader.py", line 86, in get_model
    model.load_weights(model_config.model, model_config.download_dir,
  File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 376, in load_weights
    param = params_dict[name]
KeyError: 'base_model.model.model.layers.0.self_attn.qkv_proj.lora_A.weight'

Have noticed before that models created with axolotl are sometimes not compatible with vllm. Let's shelve this for the moment.

[ggbetz]

Closing this, too.