Inference in Python requires HuggingFace Token with Local

[rovo79]

Tried to test Python inference on a local copy of coreml-stable-diffusion-xl-base, but it errors out referencing a need for a HuggingFace token. Should the token still be needed if the output-mlpackages-directory is already downloaded and local?

sw_vers: ProductName: macOS ProductVersion: 14.0 BuildVersion: 23A344

torch 2.0.1

python -m python_coreml_stable_diffusion.pipeline --prompt "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" -i coreml-stable-diffusion-xl-base/packages -o ./results --compute-unit ALL
WARNING:coremltools:scikit-learn version 1.3.1 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
WARNING:coremltools:Torch version 2.0.1 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested.
INFO:__main__:Setting random seed to 93
INFO:__main__:Initializing PyTorch pipe for reference configuration
Traceback (most recent call last):
  File "/.pyenv/versions/3.10.5/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/.pyenv/versions/3.10.5/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/coreml-stablediffusion/python_coreml_stable_diffusion/pipeline.py", line 824, in <module>
    main(args)
  File "/coreml-stablediffusion/python_coreml_stable_diffusion/pipeline.py", line 703, in main
    pytorch_pipe = SDP.from_pretrained(
  File "/.pyenv/versions/coremlsd/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 939, in from_pretrained
    cached_folder = cls.download(
  File "/.pyenv/versions/coremlsd/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1441, in download
    info = model_info(
  File "/.pyenv/versions/coremlsd/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/.pyenv/versions/coremlsd/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1686, in model_info
    headers = self._build_hf_headers(token=token)
  File "/.pyenv/versions/coremlsd/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 5719, in _build_hf_headers
    return build_hf_headers(
  File "/.pyenv/versions/coremlsd/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "/.pyenv/versions/coremlsd/lib/python3.10/site-packages/huggingface_hub/utils/_headers.py", line 121, in build_hf_headers
    token_to_send = get_token_to_send(token)
  File "/.pyenv/versions/coremlsd/lib/python3.10/site-packages/huggingface_hub/utils/_headers.py", line 153, in get_token_to_send
    raise LocalTokenNotFoundError(
huggingface_hub.utils._headers.LocalTokenNotFoundError: Token is required (`token=True`), but no token found. You need to provide a token or be logged in to Hugging Face with `huggingface-cli login` or `huggingface_hub.login`. See https://huggingface.co/settings/tokens.

[atiorh]

@rovo79 Correct, you still need to access the original (PyTorch) pipeline through Hugging Face Hub in order to initialize the rest of the pipeline that the CoreML models do not subsume

[rovo79]

I keep reading the steps but I'm sure I'm somehow misunderstanding them. Following Using Ready-made Core ML Models from Hugging Face Hub , under Step 3:

Step 3: Use git clone to download a copy of the repo that includes all model variants. For Stable Diffusion version 1.4, you'd issue the following command in your terminal:

git clone https://huggingface.co/apple/coreml-stable-diffusion-v1-4

Except in my case, I did git clone https://huggingface.co/apple/coreml-stable-diffusion-xl-base

Then it says: "If you prefer to download specific variants instead of cloning the repos, you can use the huggingface_hub Python library. "

From there, I went down to the "Image Generation with Python" section to try and run python inference on the cloned repo.

[rovo79]

@atiorh It took me a while today but was able to get python inference on sdxl to work. Initially I could only get SWIFT inference to work. I wanted to test out apple/coreml-stable-diffusion-xl-base. I had already cloned the repo to my local drive.

Command I was testing: python -m python_coreml_stable_diffusion.pipeline --prompt "a photo of an astronaut riding a horse on mars" -i models/coreml-stable-diffusion-xl-base_packages -o results --compute-unit CPU_AND_GPU --model-version apple/coreml-stable-diffusion-xl-base

Error I was getting: OSError: Error no file named model_index.json found in directory coreml-stable-diffusion-xl-base.

It clicked for me that the apple/coreml-stable-diffusion-xl-base doesn't have a _modelindex.json file, but the stabilityai/stable-diffusion-xl-base-1.0 does have one!

I then updated the prompt setting the model-version to the stability repo: python -m python_coreml_stable_diffusion.pipeline --prompt "a photo of an astronaut riding a horse on mars" -i coreml-stable-diffusion-xl-base/packages -o results --compute-unit CPU_AND_GPU --seed 93 --model-version stabilityai/stable-diffusion-xl-base-1.0

It was still having trouble determining what the package files were, so then added the flag --model-sources packages.

Another confusing piece, under "Using Ready-made Core ML Models from Hugging Face Hub" , all the links have a different href than their link text. Maybe it's done this way on purpose as sort of quick reference...

Now I have a baseline to work from when testing out the models.

[atiorh]

Thanks for the detailed walkthrough of your workflow and feedback! README can definitely be updated/improved. I will try to spend some time on it this week. However, please feel free to submit proposed changes if you get the time.