[Agents and Tools] Custom tool not showing up in the toolbox list

[sayakpaul]

I have coded an inpainter tool: https://huggingface.co/spaces/sayakpaul/inpainting-tool/

Then I load the tool as follows:

from transformers import load_tool

inpainter = load_tool("sayakpaul/inpainting-tool")

When running print(f"Description: '{inpainter.description}'"), it shows the output as expected:

Description: 'This is a tool that inpaints some parts of an image StableDiffusionInpaintPipeline according to a prompt. It takes three inputs: image, which should be the original image which will be inpainted, mask_image, which should be used to determine which parts of the original image (stored in the image variable) should be inpainted, and prompt, which should be the prompt to use to guide the inpainting process. It returns the inpainted image.'

Then, I try to add this tool to the list of existing tools:

from transformers import HfAgent

tools = [inpainter]
agent = HfAgent(
    "https://api-inference.huggingface.co/models/bigcode/starcoder",
    additional_tools=tools
)

However, the tool is not added to the toolkit (it just leaves a bullet point):

print("\n".join([f"- {a}" for a in agent.toolbox.keys()]))

- document_qa
- image_captioner
- image_qa
- image_segmenter
- transcriber
- summarizer
- text_classifier
- text_qa
- text_reader
- translator
- image_transformer
- text_downloader
- image_generator
- video_generator
-

As a result, when I run:

image = agent.run(
    "Inpaint the image: 'a cute dinosaur'", 
    image=orig_image,
    mask_image=mask_image,
    return_code=True
)

this is the code that gets generated:

==Explanation from the agent==
I will use the following  tools: `image_transformer` to transform the image, then `image_segmenter` to create a mask, then `image_transformer` to inpaint the image.

==Code generated by the agent==
prompt = "a cute dinosaur"
image = image_transformer(image, prompt)
mask = image_segmenter(image, prompt)
inpainted_image = image_transformer(image, prompt, mask)

As we can see, there's no mention of the image_inpainter here.

Anything am I missing out on?

Here's my Colab Notebook for reproduction: https://colab.research.google.com/drive/1BuNz2-7ePeaRaeI7yNc3kqDOzfdXOsUE?usp=sharing

I followed this guide during the process: https://huggingface.co/docs/transformers/custom_tools#using-custom-tools

transformers-cli env gives:

- `transformers` version: 4.29.2
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.8
- JaxLib version: 0.4.7
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

[amyeroberts]

cc @sgugger @LysandreJik

[sgugger]

Thanks for the report! I can confirm that load_tool does not properly set the name as described in the tool_config.json. One workaround is to implement it properly in the class of your tool (setting the name attribute like you did for the description) but the tool config should probably override the non-defined attribute. Will work on a fix this morning!

[sayakpaul]

Thank you!

For the custom tool, I referred to https://huggingface.co/spaces/huggingface-tools/text-to-image/blob/main/text_to_image.py#L14 and saw it didn't also assign the name member.

[sgugger]

Yes but this one is in the default tools, so is loaded differently. Bug should be fixed soon in any case.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.