Some tools cannot be used, when you cannot bypass the LLM

[SebastianStehle]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain.js documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain.js rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

I am not sure if this is a bug or a feature request. But anyway. I am using Dall-E with openai tools. My code looks similar to this:

import { AgentExecutor, createOpenAIToolsAgent } from "langchain/agents";
import { pull } from "langchain/hub";
import { ChatOpenAI } from "@langchain/openai";
import type { ChatPromptTemplate } from "@langchain/core/prompts";
import { DallEAPIWrapper } from "@langchain/openai";

// Get the prompt to use - you can modify this!
// If you want to see the prompt in full, you can at:
// https://smith.langchain.com/hub/hwchase17/openai-tools-agent
const prompt = await pull<ChatPromptTemplate>("hwchase17/openai-tools-agent");

const tool = new DallEAPIWrapper({
  n: 1, // Default
  model: "dall-e-3", // Default
  apiKey: openAIKey,
});

const tools = [tool];

const llm = new ChatOpenAI({
  model: "gpt-3.5-turbo-1106",
  temperature: 0,
  openAIApiKey: openAIKey,
});

const agent = await createOpenAIToolsAgent({
  llm,
  tools,
  prompt,
});

const agentExecutor = new AgentExecutor({
  agent,
  tools,
});

const result = await agentExecutor.invoke({
  input: "Create the image of a dog",
});

console.log(result);

Error Message and Stack Trace (if applicable)

No response

Description

Usually OpenAI shows the URL, but query params are not included and the final URL differs from the result that Dall-E provides. This seems to be some kind of Spam protection and makes several tool scenarios difficult or impossible.

I could not find a good workaround yet. My idea was to encode the URL in a way that OpenAI does not understand and then decode it later, but it would not very bullet proof. So ultimately I think there should be a way to bypass the LLM for some tool calls.

System Info

Langchain 0.1.33

[dosubot[bot]]

To address the issue with the URL discrepancy when using Dall-E through OpenAI tools in LangChain.js, you might consider implementing a custom solution to bypass the LLM for specific tool calls. This involves directly handling the API requests to Dall-E, including managing the query parameters that are missing in the URL provided by OpenAI.

Here's a conceptual approach:

1. Directly Use OpenAI API for Dall-E Requests: Implement a conditional logic in your application to directly invoke the OpenAI API for Dall-E image generation requests, bypassing the LangChain.js LLM wrapper for these calls. This allows you to manually include all necessary query parameters in the request.

2. Custom Handling for Dall-E Requests: You can add a custom method within your application or extend the OpenAI class in LangChain.js to handle Dall-E requests. This method would construct the request URL with the required query parameters and make the API call directly, ensuring the URL is correctly formatted.

Here's an example of how you might implement such a method:

async generateDallEImage(prompt: string, options?: any): Promise<any> {
  const dallEUrl = `https://api.openai.com/v1/dalle/generate?prompt=${encodeURIComponent(prompt)}&your_query_param=your_value`;

  const requestOptions = {
    headers: {
      "Authorization": `Bearer ${this.apiKey}`,
    },
  };

  try {
    const response = await fetch(dallEUrl, requestOptions);
    const data = await response.json();
    return data;
  } catch (error) {
    console.error("Error generating Dall-E image:", error);
    throw error;
  }
}

This method directly constructs the request URL, including any necessary query parameters, and uses fetch to make the HTTP request. Adjust this example based on your specific environment and requirements.

Implementing a direct API call approach allows you to bypass limitations related to URL formatting and ensures that your Dall-E requests include all necessary information.

Sources

• libs/langchain-openai/src/llms.ts
• libs/langchain-openai/src/tools/dalle.ts
• langchain/src/agents/executor.ts

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.