Workflow example not working with Azure-OpenAI

[titanlogic]

Describe the bug When working through the Workflow course: https://learn.griptape.ai/courses/compare-movies-workflow/, I am unable to complete it once I arrive at the section adding the ToolkitTasks for WebScraper and TaskMemoryClient.

I only have Azure OpenAI at my disposal and even though I have specified a PromptDriver pointing to our Azure instance and required values at every level I can think of (WorkFlow, PromptTask, and ToolkitTask), when I run the code the ToolkitTask always tries to hit openai rather than Azure openai. I was able to successfully complete the chatbot course using PromptDriver pointing to our Azure instance, but no luck here. The other portions of the WorkFlow example work just fine with our Azure instance, but ToolkitTask generates the following errors when run:

WARNING:root:<RetryCallState 2894334457168: attempt #2; slept for 2.0; last result: failed (AuthenticationError Error code: 401 - {'error': {'message': 'Incorrect API key provided: . You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}})>

It seems some part of the code is hardcoded to use openai rather than a PromptDriver redirect, but I can't find where.

To Reproduce Steps to reproduce the behavior:

1. Use a PromptDriver in this mode to attempt to point to an Azure OpenAI instance: prompt_driver=AzureOpenAiChatPromptDriver( model="gpt-4", azure_deployment=os.environ["AZURE_OPENAI_4_1106Preview_DEPLOYMENT_ID"], azure_endpoint=os.environ["AZURE_OPENAI_API_BASE"], api_key=os.environ["OPENAI_API_KEY"], api_version=os.environ["AZURE_OPENAI_API_VERSION"], base_url=os.environ["AZURE_OPENAI_API_BASE"] )

Expected behavior The workflow will run normally and be able to call out to metacritic to retrieve movie descriptions.

Screenshots If applicable, add screenshots to help explain your problem. N/A

Desktop (please complete the following information):

• OS: Windows 11
• Browser: Edge / Chrome
• Version 119.0.2151.72 / 119.0.6045.160

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context Add any other context about the problem here.

[shhlife]

Thanks for the report! I'll look into it.

[shhlife]

I've done some experiments today, and I'm able to reproduce your issue. I'm checking with the team to see if it's a known issue! will report back! Thanks again!

[shhlife]

So part of the problem might be that the embedding_driver needs to be set as well as the prompt_driver.

First, I created an azure_driver and an azure_embedding_driver:

azure_driver = AzureOpenAiChatPromptDriver(
    api_key=AZURE_OPENAI_API_KEY_2,
    model="gpt-4",
    azure_deployment=AZURE_OPENAI_4_DEPLOYMENT_ID,
    azure_endpoint=AZURE_OPENAI_API_BASE_2,
)

azure_embedding_driver = AzureOpenAiEmbeddingDriver(
    api_key=AZURE_OPENAI_API_KEY_2,
    model="text-embedding-ada-002",
    azure_deployment=AZURE_OPENAI_4_DEPLOYMENT_ID,
    azure_endpoint=AZURE_OPENAI_API_BASE_2,
)

Then, I created a simple pipeline structure specifying both prompt_driver and embedding_driver:

pipeline = Pipeline(prompt_driver=azure_driver, embedding_driver=azure_driver)

pipeline.add_task(
    ToolkitTask(
        "{{ args[0]}}", prompt_driver=azure_driver, tools=[WebScraper(off_prompt=False)]
    )
)

pipeline.run("What's https://griptape.ai all about?")

Please give this a try and see if it works for you?

More info: https://docs.griptape.ai/en/latest/griptape-framework/data/embedding-drivers/#override-default-structure-embedding-driver

Cheers! -Jason

[titanlogic]

I sure will Jason, thank you so much!

[titanlogic]

Worked like a charm Jason, thank you so much. I'm a little embarrassed, looking at the toolset I should have realized there might be opaque embeddings in the workflow lol.

Looking forward to watching this project grow, have a great day!

[shhlife]

Awesome, I'm so glad it worked out! And don't be embarrassed at all - I didn't know about the embeddings driver either! Much thanks to @collindutter for pointing it out to me. :)

-Jason