[Agents] RuntimeError: Invalid device string: 'hakurei/waifu-diffusion'

simonSlamka commented 1 year ago

System Info

Hey!

I have encountered an issue in the agents extra where calling load_tool() with model_repo_id=<sd checkpoint model id> causes a RuntimeError to occur. It would appear that the repo id is being used as a device_id when using the text-to-image tool:

`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
Traceback (most recent call last):
  File "/home/simtoon/transformers/main.py", line 18, in <module>
    img = imggen("cute anime cat")
  File "/home/simtoon/.cache/huggingface/modules/transformers_modules/huggingface-tools/text-to-image/8a3d5357ffa541880148f2425c83ba89f7d56172/text_to_image.py", line 45, in __call__
    self.setup()
  File "/home/simtoon/.cache/huggingface/modules/transformers_modules/huggingface-tools/text-to-image/8a3d5357ffa541880148f2425c83ba89f7d56172/text_to_image.py", line 36, in setup
    self.pipeline.to(self.device)
  File "/home/simtoon/transformers/venv/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 682, in to
    module.to(torch_device, torch_dtype)
  File "/home/simtoon/transformers/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1126, in to
    device, dtype, non_blocking, convert_to_format = torch._C._nn._parse_to(*args, **kwargs)
RuntimeError: Invalid device string: 'hakurei/waifu-diffusion'

- `transformers` version: 4.30.2
- Platform: Linux-5.4.0-139-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A

Who can help?

cc @Narsil

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

install transformers
install transformers[agents]
call load_tool with model_repo_id set to a valid sd checkpoint repo id: imggen = load_tool(task_or_repo_id="huggingface-tools/text-to-image", model_repo_id="hakurei/waifu-diffusion") ... img = imggen("cute anime cat")
observe the exception

Expected behavior

Pull model from hf hub or load from local and use in the txt2img task

Narsil commented 1 year ago

Device_string is not looking good.

@LysandreJik might know better than me where this is coming from.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

simonSlamka commented 1 year ago

@LysandreJik afaik this still hasn't been fixed

LysandreJik commented 1 year ago

Hey @simonSlamka sorry for the late response. I think what you're trying to do is out of scope of what we're trying to do, we haven't designed the remote tools to work by specifying specific checkpoints in this way.

For this tool in particular (but this will need to be adapted to remote tools as these can be community contributed), here's how you would go about it:

from transformers import load_tool

imggen = load_tool(task_or_repo_id="huggingface-tools/text-to-image")
imggen.default_checkpoint = "hakurei/waifu-diffusion"

img = imggen("cute anime cat")

I think the best way to leverage a given checkpoint here would be to clone the existing remote tool, replace the checkpoint, and update the generation settings so that they work best with the checkpoint you have in mind. So you would have a remote tool of your own, for example simonSlamka/anime-text-to-image that could be used to replace the existing image generation tool in the toolbox (or provided as an additional tool with anime as its focus).

Hope that helps!

simonSlamka commented 1 year ago

Hi,

Thanks a lot for your assistance and guidance. I will do what you suggested.

Have a nice day!

huggingface / transformers