Streaming Agent Error -----> TypeError: 'function' object is not iterable

[getshaun24]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain rather than my code.

Example Code

from flask import Blueprint, request, Response, current_app, jsonify, stream_with_context
from flask_jwt_extended import jwt_required, current_user
from project import db
from langchain_community.llms import Ollama
from langchain_community.vectorstores import Chroma
from langchain_community.document_loaders.csv_loader import CSVLoader
from langchain_community.embeddings import OllamaEmbeddings
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder, PromptTemplate
from langchain_community.chat_message_histories import MongoDBChatMessageHistory
from langchain.agents import tool, AgentExecutor, create_react_agent
from langchain.tools.retriever import create_retriever_tool
from langchain.agents import AgentExecutor
from langchain_openai import OpenAI
from langchain_openai import OpenAIEmbeddings
import json
from langchain_core.utils.function_calling import convert_to_openai_tool
from langchain_experimental.llms.ollama_functions import OllamaFunctions
from langchain_openai import ChatOpenAI
from langchain.callbacks.streaming_stdout_final_only import FinalStreamingStdOutCallbackHandler
import time
import asyncio
from random import choice

chat_bot_2_Blueprint = Blueprint('chat_bot_2', __name__)
@chat_bot_2_Blueprint.route('/api/ai/chat_bot_2/', methods=['GET', 'POST']) # <- from '/'
@jwt_required()
async def chat_bot_2():

    user_ID = current_user.user_ID
    chat_input = request.json.get("chat_input", None)
    chat_ID = request.json.get("live_chat_ID", None)

    path_to_chroma_db = "./project/api/ai/chroma_db/"
    path_to_csv = "/love_qa.csv"

    loader = CSVLoader(file_path=path_to_csv)
    data = loader.load()

    vector_collection = "dr_love_4"
    model_type = "open_ai"

    if model_type == "open_ai":
        embedding_model = OpenAIEmbeddings(openai_api_key=current_app.config['OPENAI_SECRET'])
        llm = ChatOpenAI(
            openai_api_key=current_app.config['OPENAI_SECRET'],
            model_name="gpt-3.5-turbo-0125",
            streaming=True,
            max_tokens=2000,
            callbacks=[FinalStreamingStdOutCallbackHandler(answer_prefix_tokens=['Final', ' Answer', ':'])],)
    else:
        model_type = OllamaEmbeddings(model="llama2")
        llm = OllamaFunctions(model="llama2")
        print('LLAMA')

    react_template = get_react_template()
    custom_react_prompt = PromptTemplate.from_template(react_template)

    # save to disk
    #vectorstore = Chroma.from_documents(documents=data, embedding=embedding_model, persist_directory=path_to_chroma_db + vector_collection)

    # load from disk
    vectorstore = Chroma(persist_directory=path_to_chroma_db + vector_collection, embedding_function=embedding_model)
    retriever = vectorstore.as_retriever(search_type="similarity", search_kwargs={"k": 2})

    retreiver_title = "hormone_doctor_medical_questions_and_answers"
    retreiver_description = "Searches and returns a series of questions and answers on hormones and medical advice and also personal questions"
    retriever_tool = create_retriever_tool(retriever, retreiver_title, retreiver_description)

    tools = [confetti, get_word_length, retriever_tool]

    message_history = MongoDBChatMessageHistory(session_id=chat_ID, connection_string=current_app.config['MONGO_CLI'], database_name=current_app.config['MONGO_DB'], collection_name="chat_histories")

    agent = create_react_agent(llm=llm, tools=tools, prompt=custom_react_prompt)
    agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, max_iterations=5, max_execution_time=100, return_intermediate_steps=True, handle_parsing_errors=True).with_config({"run_name": "Agent"})

    async def stream_output(chat_input):

        # async for chunk in agent_executor.stream({"input": chat_input, "chat_history": message_history}):
        #     print(chunk)
        #     if "output" in chunk:
        #         # print(f'Final Output: {chunk["output"]}')
        #         yield chunk["output"]

        async for event in agent_executor.astream_events({"input": chat_input, "chat_history": message_history}, version="v1"):
            kind = event["event"]
            if kind == "on_chain_start":
                if (event["name"] == "Agent"):  # Was assigned when creating the agent with `.with_config({"run_name": "Agent"})`
                    print(f"Starting agent: {event['name']} with input: {event['data'].get('input')}")
            elif kind == "on_chat_model_stream":
                content = event["data"]["chunk"].content
                if content:
                    print(content, end="|")
                    yield content

        update_chat_history(user_ID, chat_ID, chat_input)

    return Response(stream_with_context(stream_output(chat_input)), mimetype='text/event-stream')

def update_chat_history(user_ID, chat_ID, chat_input):
    preview = None
    chats = current_user.chats
    # Assume chat_ID and user_ID are defined earlier in your code
    chat_exists = any(chat_ID == item.get('chat_ID') for item in chats)
    if not chat_exists:
        db.credentials.update_one({"user_ID": user_ID}, {"$push": {'chats': {'chat_ID': chat_ID, 'preview': chat_input}}})
    else:
        #Move the items to the front of the list
        #get the preview
        for item in chats:
            if item['chat_ID'] == chat_ID:
                preview = item['preview']

    # First, pull the item if it exists
    db.credentials.update_one({"user_ID": user_ID, "chats.chat_ID": chat_ID}, {"$pull": {"chats": {"chat_ID": chat_ID}}})
        # Then, push the item to the array
    db.credentials.update_one({"user_ID": user_ID}, {"$push": {"chats": {"chat_ID": chat_ID, "preview": preview}}})

def get_react_template():

    react_template="""
                Assistant is a large language model trained by OpenAI.

                Assistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.

                Assistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.

                Overall, Assistant is a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist.

                TOOLS:
                ------

                Assistant has access to the following tools:

                {tools}

                To use a tool, please use the following format:

                Thought: Do I need to use a tool? Yes
                Action: the action to take, should be one of [{tool_names}]
                Action Input: the input to the action
                Observation: the result of the action

                When you have a response to say to the Human, or if you do not need to use a tool, you MUST use the format:

                Thought: Do I need to use a tool? No
                Final Answer: [your response here]

                Begin!

                Previous conversation history:
                {chat_history}

                Question: {input}

                New input:
                {agent_scratchpad}
        """
    return react_template

@tool
async def confetti(string: str) -> str:
    """Adds a random word to the string"""
    return "confetti " + string

@tool
async def get_word_length(word: str) -> int:
    """Returns the length of a word."""
    return len(word)

@tool
async def where_cat_is_hiding() -> str:
    """Where is the cat hiding right now?"""
    return choice(["under the bed", "on the shelf", "in your heart"])

Error Message and Stack Trace (if applicable)

Debugging middleware caught exception in streamed response at a point where response headers were already sent. Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/werkzeug/wsgi.py", line 500, in next return self._next() ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/werkzeug/wrappers/response.py", line 50, in _iter_encoded for item in iterable: TypeError: 'function' object is not iterable

Description

I am trying to stream the output from an OpenAI agent using astream or astream_events. Neither work.

I have tried many different potential solutions and would appreciate some guidance.

Thank You.

System Info

System Information

OS: Darwin OS Version: Darwin Kernel Version 23.3.0: Wed Dec 20 21:33:31 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T8112 Python Version: 3.11.2 (main, Mar 24 2023, 00:31:37) [Clang 14.0.0 (clang-1400.0.29.202)]

Package Information

langchain_core: 0.1.23 langchain: 0.1.7 langchain_community: 0.0.20 langsmith: 0.0.87 langchain_cli: 0.0.21 langchain_experimental: 0.0.51 langchain_openai: 0.0.6 langchainhub: 0.1.14 langserve: 0.0.41

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph

[dosubot[bot]]

🤖

Hey @getshaun24! Great to see you diving into another challenge. Hope you're ready for some fun troubleshooting together! 🚀

Based on the error message you provided, it seems like the issue is related to the stream_output function. The error 'TypeError: 'function' object is not iterable' typically occurs when you try to iterate over a function object instead of calling it. In the context of LangChain and the provided code, it seems like you might be trying to iterate over the aiter method of the AsyncIteratorCallbackHandler class without calling it.

To resolve this issue, you should call the aiter method and then iterate over its result. Here's an example of how you can do this:

handler = AsyncIteratorCallbackHandler()
# Initialize and start your OpenAI agent here
# ...

# Call the aiter method and get the async iterator
aiter = handler.aiter()

# Now you can iterate over the result
async for token in aiter:
    print(token)

In this example, handler.aiter() is called to get an async iterator, and then async for is used to iterate over the tokens produced by the OpenAI agent.

You can find more details about this in the libs/langchain/langchain/callbacks/streaming_aiter.py file in the LangChain repository.

Regarding the stream_output function in your code, it seems to be an asynchronous generator function that is intended to yield the output of the OpenAI agent in a streaming manner. However, the error message suggests that this function is being treated as an iterable object, which is causing the TypeError.

Please ensure that you are correctly using the stream_output function as an asynchronous generator. You should be using the async for keyword to iterate over its results, like so:

async for output in stream_output(chat_input):
    # process output here

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• openai tools don't work with streaming=True
• libs/langchain/langchain/callbacks/streaming_aiter.py
• libs/langchain/langchain/schema/callbacks/streaming_stdout.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[getshaun24]

This was not helpful to me, please can you try and outline a more complete answer. Thank you.

[prnvsharma4]

hey @getshaun24 were you able to resolve this?

[eyurtsev]

Hi all, just a reminder that if you a person to take a look at the code try to reduce it to the minimal reproducible example that fails. This code has a lot of stuff, so it's impossible to tell whether this is a bug report or an issue in user code .

This is something that's better sent to discussions for Q&A until it's possible to localize the actual issue.

If you can get it to a point where someone can just copy and paste the code into their notebooks and the code is not more than ~10 lines of code that's a great bug report!

Closing as we're unlikely to review the issue as is