Framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.
I have encountered an error while trying to use the VisionTool(), while I'm trying to use AzureAI with the vision tool, but it wrongly mapped to the OpenAI API,
can we use AzureOpenAI in the vision tool?
error:
I encountered an error while trying to use the tool. This was the error: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable.
Tool Vision Tool accepts these inputs: Vision Tool(image_path_url: 'string') - This tool uses OpenAI's Vision API to describe the contents of an image.
from crewai import Agent, Crew, Task
from langchain_openai import AzureChatOpenAI
from openai import AzureOpenAI, OpenAI
from pdf2image import convert_from_path
from pydantic import BaseModel
from typing import List
from textwrap import dedent
from crewai_tools import VisionTool
import os
I encountered an error while trying to use the tool. This was the error: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable.
Tool Vision Tool accepts these inputs: Vision Tool(image_path_url: 'string') - This tool uses OpenAI's Vision API to describe the contents of an image.
Description
I have encountered an error while trying to use the VisionTool(), while I'm trying to use AzureAI with the vision tool, but it wrongly mapped to the OpenAI API,
can we use AzureOpenAI in the vision tool?
error:
I encountered an error while trying to use the tool. This was the error: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable. Tool Vision Tool accepts these inputs: Vision Tool(image_path_url: 'string') - This tool uses OpenAI's Vision API to describe the contents of an image.
vision_tool = VisionTool( config= { "llm": { "provider": "azure_openai", "config": { "api_key": os.getenv('AZURE_OPENAI_API_KEY'), "model": os.getenv('AZURE_MODEL'), "deployment_name": os.getenv('AZURE_MODEL') } } } )
Steps to Reproduce
use the above code with the AzureAI credits.
Expected behavior
Run with AzureAI
Screenshots/Code snippets
from crewai import Agent, Crew, Task from langchain_openai import AzureChatOpenAI from openai import AzureOpenAI, OpenAI from pdf2image import convert_from_path from pydantic import BaseModel from typing import List from textwrap import dedent from crewai_tools import VisionTool import os
vision_tool = VisionTool( config= { "llm": { "provider": "azure_openai", "config": { "api_key": os.getenv('AZURE_OPENAI_API_KEY'), "model": os.getenv('AZURE_MODEL'), "deployment_name": os.getenv('AZURE_MODEL') } } } )
def Document_content_analyzer_agent(path_of_page): agent = Agent( role="Document Content Analyst", goal=dedent(""" ... """),
backstory=( """... ... ... ... """ ), allow_delegation=False, tools=[vision_tool], llm=azure_llm,
llm=LLM_GPT4o,
verbose=True, memory=False)
task = Task( description=dedent(f""" page_path = {path_of_page} ... ... ... ... """), expected_output=""" "figures,charts,flowchart,plots and tables informations only." """, agent=agent, human_input=False)
crew_results_editor = Crew( agents=[agent], tasks=[task], verbose=True ) output_editor = crew_results_editor.kickoff() return output_editor.raw
Operating System
Ubuntu 20.04
Python Version
3.10
crewAI Version
0.41.1
crewAI Tools Version
0.13.2
Virtual Environment
Venv
Evidence
I encountered an error while trying to use the tool. This was the error: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable. Tool Vision Tool accepts these inputs: Vision Tool(image_path_url: 'string') - This tool uses OpenAI's Vision API to describe the contents of an image.
vision_tool = VisionTool( config= { "llm": { "provider": "azure_openai", "config": { "api_key": os.getenv('AZURE_OPENAI_API_KEY'), "model": os.getenv('AZURE_MODEL'), "deployment_name": os.getenv('AZURE_MODEL') } } } )
Possible Solution
Currently No
Additional context
NA