Evaluate: nvidia/nemotron-3-8b-XXX

[ggbetz]

Check upon issue creation:

• [x] The model has not been evaluated yet and doesn't show up on the CoT Leaderboard.
• [x] There is no evaluation request issue for the model in the repo.
• [x] The parameters below have been adapted and shall be used.

Parameters (for XXX in ["base-4k", "chat-4k-sft", "chat-4k-rlhf", "chat-4k-steerlm"]):

NEXT_MODEL_PATH=nvidia/nemotron-3-8b-XXX
NEXT_MODEL_REVISION=main
NEXT_MODEL_PRECISION=float16
MAX_LENGTH=2048 
GPU_MEMORY_UTILIZATION=0.8
VLLM_SWAP_SPACE=12

ToDos:

• [ ] Wait for VLLM support / ported model
• [ ] Run cot-eval pipeline
• [ ] Merge pull requests for cot-eval results datatset (> @ggbetz)
• [ ] Create eval request record to update metadata on leaderboard (> @ggbetz)

[yakazimir]

Seems to be a missing dependency:

2024-05-10T16:58:23.246848366Z 
2024-05-10T16:58:23.246851786Z The above exception was the direct cause of the following exception:
2024-05-10T16:58:23.246855336Z 
2024-05-10T16:58:23.246858436Z Traceback (most recent call last):
2024-05-10T16:58:23.246862106Z   File "/usr/local/bin/cot-eval", line 8, in <module>
2024-05-10T16:58:23.246866196Z     sys.exit(main())
2024-05-10T16:58:23.246869416Z   File "/workspace/cot-eval/src/cot_eval/__main__.py", line 149, in main
2024-05-10T16:58:23.246886096Z     llm = VLLM(
2024-05-10T16:58:23.246889666Z   File "/usr/local/lib/python3.10/dist-packages/langchain_core/load/serializable.py", line 120, in __init__
2024-05-10T16:58:23.246960076Z     super().__init__(**kwargs)
2024-05-10T16:58:23.246975396Z   File "/usr/local/lib/python3.10/dist-packages/pydantic/v1/main.py", line 339, in __init__
2024-05-10T16:58:23.247208935Z     values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
2024-05-10T16:58:23.247257415Z   File "/usr/local/lib/python3.10/dist-packages/pydantic/v1/main.py", line 1102, in validate_model
2024-05-10T16:58:23.247768653Z     values = validator(cls_, values)
2024-05-10T16:58:23.247818133Z   File "/usr/local/lib/python3.10/dist-packages/langchain_community/llms/vllm.py", line 88, in validate_environment
2024-05-10T16:58:23.247822213Z     values["client"] = VLLModel(
2024-05-10T16:58:23.247825443Z   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py", line 112, in __init__
2024-05-10T16:58:23.247828423Z     self.llm_engine = LLMEngine.from_engine_args(
2024-05-10T16:58:23.247831053Z   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 174, in from_engine_args
2024-05-10T16:58:23.247893782Z     engine_configs = engine_args.create_engine_configs()
2024-05-10T16:58:23.247938782Z   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py", line 390, in create_engine_configs
2024-05-10T16:58:23.248127661Z     model_config = ModelConfig(
2024-05-10T16:58:23.248149451Z   File "/usr/local/lib/python3.10/dist-packages/vllm/config.py", line 121, in __init__
2024-05-10T16:58:23.248154331Z     self.hf_config = get_config(self.model, trust_remote_code, revision,
2024-05-10T16:58:23.248158941Z   File "/usr/local/lib/python3.10/dist-packages/vllm/transformers_utils/config.py", line 22, in get_config
2024-05-10T16:58:23.248275101Z     config = AutoConfig.from_pretrained(
2024-05-10T16:58:23.248306351Z   File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 928, in from_pretrained
2024-05-10T16:58:23.248973368Z     config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
2024-05-10T16:58:23.249007878Z   File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 631, in get_config_dict
2024-05-10T16:58:23.249013228Z     config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
2024-05-10T16:58:23.249024398Z   File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 686, in _get_config_dict
2024-05-10T16:58:23.249332667Z     resolved_config_file = cached_file(
2024-05-10T16:58:23.249383937Z   File "/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py", line 452, in cached_file
2024-05-10T16:58:23.249554396Z     raise EnvironmentError(
2024-05-10T16:58:23.249572106Z OSError: nvidia/nemotron-3-8b-base-4k does not appear to have a file named config.json. Checkout 'https://huggingface.co/nvidia/nemotron-3-8b-base-4k/main' for available files.```

[ggbetz]

Sorry, I just realize that this won't run with VLLM, requires NVDIA's nemo framework: NeMo inference

I'll close this