search

Lightning-AI / litgpt

Pretrain, finetune, deploy 20+ LLMs on your own data. Uses state-of-the-art techniques: flash attention, FSDP, 4-bit, LoRA, and more.
https://lightning.ai
Apache License 2.0
6.85k stars 726 forks source link

The `litgpt evaluate` command attempts to download config files from gated repos #1349

Closed awaelchli closed 3 weeks ago

awaelchli commented 3 weeks ago

If you want to evaluate a checkpoint that you previously downloaded, you may run into the issue that the checkpoint is of a model that is gated on huggingface:

litgpt evaluate --checkpoint_dir checkpoints/google/gemma-7b/ --out_dir out/evaluation --tasks mmlu --limit 10
2024-04-24:13:02:43,993 INFO     [huggingface.py:162] Using device 'cuda'
Traceback (most recent call last):
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/google/gemma-7b/resolve/main/config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/transformers/utils/hub.py", line 398, in cached_file
    resolved_file = hf_hub_download(
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1403, in hf_hub_download
    raise head_call_error
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1261, in hf_hub_download
    metadata = get_hf_file_metadata(
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1674, in get_hf_file_metadata
    r = _request_wrapper(
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 369, in _request_wrapper
    response = _request_wrapper(
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 393, in _request_wrapper
    hf_raise_for_status(response)
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 321, in hf_raise_for_status
    raise GatedRepoError(message, response) from e
huggingface_hub.utils._errors.GatedRepoError: 401 Client Error. (Request ID: Root=1-66293b34-73019ffb2df5e5921ce33e51;32481442-1f5b-4114-b085-a8a7af4c710c)

Cannot access gated repo for url https://huggingface.co/google/gemma-7b/resolve/main/config.json.
Access to model google/gemma-7b is restricted. You must be authenticated to access it.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/adrian/.conda/envs/lit-gpt/bin/litgpt", line 8, in <module>
    sys.exit(main())
  File "/home/adrian/repositories/lit-gpt/litgpt/__main__.py", line 143, in main
    fn(**kwargs)
  File "/home/adrian/repositories/lit-gpt/litgpt/eval/evaluate.py", line 102, in convert_and_evaluate
    model = HFLM(repo_id, state_dict=state_dict, device=device, batch_size=batch_size, dtype=dtype, local_files_only=True)
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/lm_eval/models/huggingface.py", line 188, in __init__
    self._get_config(
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/lm_eval/models/huggingface.py", line 453, in _get_config
    self._config = transformers.AutoConfig.from_pretrained(
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 928, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/transformers/configuration_utils.py", line 631, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/transformers/configuration_utils.py", line 686, in _get_config_dict
    resolved_config_file = cached_file(
  File "/home/adrian/.conda/envs/lit-gpt/lib/python3.10/site-packages/transformers/utils/hub.py", line 416, in cached_file
    raise EnvironmentError(
OSError: You are trying to access a gated repo.
Make sure to have access to it at https://huggingface.co/google/gemma-7b.
401 Client Error. (Request ID: Root=1-66293b34-73019ffb2df5e5921ce33e51;32481442-1f5b-4114-b085-a8a7af4c710c)

Cannot access gated repo for url https://huggingface.co/google/gemma-7b/resolve/main/config.json.
Access to model google/gemma-7b is restricted. You must be authenticated to access it.

This happens because the eval harness wants the config file and goes through HF to download it: https://github.com/EleutherAI/lm-evaluation-harness/blob/3196e907fa195b684470a913c7235ed7f08a4383/lm_eval/models/huggingface.py#L190-L194

This makes sense if the config file is not already downloaded, but in our case it is. So it is pointless to try to download it. I need to find a way to work around this because my token doesn't have access to all models. Probably needs to be addressed somehow upsream.

rasbt commented 3 weeks ago

Hm yeah, this sounds annoying. In an ideal world, they would check for the file locally before attempting to download. Not sure, but could be wortwhile raising this issue in the harness repo