search

huggingface / huggingface_hub

The official Python client for the Huggingface Hub.
https://huggingface.co/docs/huggingface_hub
Apache License 2.0
2.14k stars 559 forks source link

vague invalid credentials errror with api key that works locally but not on cloud server #1616

Closed bazylhorsey closed 1 year ago

bazylhorsey commented 1 year ago

System Info

transformers = "^4.31.0" python = "3.10" Running on elastic beanstalk ec2

Who can help?

@Narsil

Information

Tasks

Reproduction

  1. create a private model that is private.
  2. start it with pipeline
  3. use valid and invalid use_auth_token field with user token

  4. play with firewall and watch behavior

    class PredictTransformersPipelineTask(Task):
"""
Abstraction of Celery's Task class to support loading transformers model.
"""

task_name = ""
model_name = ""
abstract = True

def __init__(self):
    super().__init__()
    self.pipeline = None

def __call__(self, *args, **kwargs):
    """
    Load pipeline on first call (i.e. first task processed)
    Avoids the need to load pipeline on each task request
    """
    if not self.pipeline:
        logging.info("Loading pipeline...")
        self.pipeline = pipeline(self.task_name, model=self.model_name, use_auth_token=settings.HUGGINGFACE_API_KEY)
        logging.info("Pipeline loaded")

    return self.run(*args, **kwargs)

    On local it works as expected, and only valid api keys download the repo. We are using a private repo named arvist/scanner-basic, which used to be named arvist/ppe-test. We noticed its strange in the error stack that it shows arvist/ppe-test for what was attempted to load in even though arvist/ppe-test is mentioned nowhere in our current deployment.

However, on cloud with correct credentials we get this error which is the same as having the wrong credentials on local:

Invalid username or password.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 677, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/code/app/main.py", line 85, in lifespan
    "object_detection": pipeline("object-detection", model="arvist/ppe-test"),
  File "/usr/local/lib/python3.10/site-packages/transformers/pipelines/__init__.py", line 705, in pipeline
    config = AutoConfig.from_pretrained(model, _from_pipeline=task, **hub_kwargs, **model_kwargs)
  File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 983, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/transformers/configuration_utils.py", line 617, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/transformers/configuration_utils.py", line 672, in _get_config_dict
    resolved_config_file = cached_file(
  File "/usr/local/lib/python3.10/site-packages/transformers/utils/hub.py", line 433, in cached_file
    raise EnvironmentError(
OSError: arvist/ppe-test is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.

[2023-08-21 21:27:14 +0000] [37] [ERROR] Application startup failed. Exiting.
[2023-08-21 21:27:14 +0000] [37] [INFO] Worker exiting (pid: 37)
[2023-08-21 21:27:14 +0000] [47] [ERROR] Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 261, in hf_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/arvist/ppe-test/resolve/main/config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/transformers/utils/hub.py", line 417, in cached_file
    resolved_file = hf_hub_download(
  File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1195, in hf_hub_download
    metadata = get_hf_file_metadata(
  File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1541, in get_hf_file_metadata
    hf_raise_for_status(r)
  File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 293, in hf_raise_for_status
    raise RepositoryNotFoundError(message, response) from e
huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-64e3d6b2-74fc632a13152e76720336e5;2a65daf3-b67e-4b8e-a12d-5b61d0dae42e)

Repository Not Found for url: https://huggingface.co/arvist/ppe-test/resolve/main/config.json.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 677, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "/usr/local/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/code/app/main.py", line 85, in lifespan
    "object_detection": pipeline("object-detection", model="arvist/ppe-test"),
  File "/usr/local/lib/python3.10/site-packages/transformers/pipelines/__init__.py", line 705, in pipeline
    config = AutoConfig.from_pretrained(model, _from_pipeline=task, **hub_kwargs, **model_kwargs)
  File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 983, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/transformers/configuration_utils.py", line 617, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/transformers/configuration_utils.py", line 672, in _get_config_dict
    resolved_config_file = cached_file(
  File "/usr/local/lib/python3.10/site-packages/transformers/utils/hub.py", line 433, in cached_file
    raise EnvironmentError(
OSError: arvist/ppe-test is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.

We think perhaps this is a firewall issue, but there is no mention of what traffic rules are required, we tried inbound 443 TCP with no changes, is the git port used at all here? It seems all that is used is the requests library which is http-based.

Expected behavior

Client should be able to download on cloud system, or firewall details should be documented.

Our use case involves downloading huggingface repos at server boot and when they're needed followed by inference/predictions.

Local and cloud behavior should be consistent using the same key in web-based inferencing services.

ArthurZucker commented 1 year ago

I'll transfer this over to huggingface_hub cc @Wauplin

Wauplin commented 1 year ago

Hi @bazylhorsey sorry for the late response, I was off for some time. Looking at the errors, I am not sure that it is a firewall issue since you are getting a proper response from the server (Repository Not Found for url: https://huggingface.co/arvist/ppe-test/resolve/main/config.json.). My first guess would be that this repo is private but you're not sending the access token.

Could you try a few things on your cloud cluster:

  1. Run huggingface-cli login to login. This will save your access token on the disk of the machine so be careful if it's a shared one.
  2. Run huggingface-cli whoami to check token is valid. It should print your username.
  3. Run 
    >>> from huggingface_hub import hf_hub_download
>>> hf_hub_download('arvist/ppe-test', 'config.json')

    To try download the file manually. This is what transformers is doing under the hood. 3.bis. if you don't want to login on your cloud machine, run

    >>> from huggingface_hub import hf_hub_download
>>> hf_hub_download('arvist/ppe-test', 'config.json', token='hf_****')
  4. If 3. worked, just run your full script again and it should work.

If any step fails, please let me know. Run huggingface-cli env and copy-paste the output in this issue, that could help found out the issue.

Hope this will help, Cheers, Wauplin :hugs: