Playwright Browser Freezing

CurtMRosenbladWheeler commented 6 months ago

System Info

LangChain Version: 0.0.354

Platform: MacOS Sonoma 14.2.1 Python Version: 3.11.6

Who can help?

@hwchase17 @agola11

Information

[X] The official example notebooks/scripts
[X] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[X] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction

Code sample that utilizes the Playwright Toolkit: https://python.langchain.com/docs/integrations/toolkits/playwright

For instance, when you attempt to run:

await navigate_tool.arun(
    {"url": "https://web.archive.org/web/20230428131116/https://www.cnn.com/world"}
)

Nothing happens after 20 minutes.

The only way I have been able to get a response was by waiting a few seconds and then turning off my computer's wifi, which would return the expected:

'Navigating to https://web.archive.org/web/20230428131116/https://www.cnn.com/world returned status code 200'

I am utilizing the browser in an agent, and here is the code:

import asyncio
from typing import Type, Optional

from langchain.agents import AgentExecutor
from langchain.schema import SystemMessage, HumanMessage, AIMessage
from long-chain.agents.format_scratchpad.openai_functions import format_to_openai_function_messages
from langchain.agents.output_parsers.openai_functions import OpenAIFunctionsAgentOutputParser
from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.tools.convert_to_openai import format_tool_to_openai_function

from langchain_community.tools.playwright.utils import create_async_playwright_browser
from langchain.tools.tavily_search import TavilySearchResults
from langchain.utilities.tavily_search import TavilySearchAPIWrapper
from langchain_community.agent_toolkits.playwright import PlayWrightBrowserToolkit
from langchain.callbacks.manager import CallbackManagerForToolRun
from langchain.pydantic_v1 import BaseModel, Field
from langchain.tools import BaseTool

from auto_sentry.src.chat_models.enums.OpenAI import OpenAI as openai_enums
from auto_sentry.src.chat_models.Memory import Memory
from auto_sentry.src.chat_models.OpenAI import GPT

class AIQuestionInput(BaseModel):
    query: str = Field(description="should contain your question for the user including specific aspects or details from the user's input that require further clarification or elaboration")

class AIQuestionTool(BaseTool):
    name = "ai_question"
    description = "useful for when you need to clairify user requests or when you need to ask them a question"
    args_schema: Type[BaseModel] = AIQuestionInput

    def _run(
        self, query: str, run_manager: Optional[CallbackManagerForToolRun] = None
    ) -> str:
        """Use the tool."""
        return query

llm = GPT(model=openai_enums.GPT_4_1106_PREVIEW)
memory = Memory("Test_Conversation")
memory.clear()

search = [TavilySearchResults(api_wrapper=TavilySearchAPIWrapper(), handle_tool_error=True, verbose=True)]

playwright_async_browser = create_async_playwright_browser(headless=True)
toolkit = PlayWrightBrowserToolkit.from_browser(async_browser=playwright_async_browser)
playwright_tools = toolkit.get_tools()

tools:list = search + playwright_tools

MEMORY_KEY = "chat_history"
prompt = ChatPromptTemplate.from_messages(
    [
        SystemMessage(
            content="You are a helpful assistant."
        ),
        MessagesPlaceholder(variable_name=MEMORY_KEY),
        HumanMessage(content="{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ]
)

memory.add(prompt.messages[0])

functions=[format_tool_to_openai_function(tool) for tool in tools]

#"""
llm_with_tools = llm.llm.bind(
    functions=functions
)

agent = (
    {
        "input": lambda x: x["input"],
        "agent_scratchpad": lambda x: format_to_openai_function_messages(
            x["intermediate_steps"]
        ),
        "chat_history": lambda x: x["chat_history"],
    }
    | prompt
    | llm_with_tools
    | OpenAIFunctionsAgentOutputParser()
)

agent_executer = AgentExecutor(
    agent=agent, tools=tools, verbose=True, return_intermediate_steps=True
)

query = HumanMessage(content=input("> "))
memory.add(query)

async def run_agent():
    print(memory.memory())
    response = await agent_executer.ainvoke(
        {"input": query, MEMORY_KEY: memory.memory()}
    )
    return response

response = asyncio.run(run_agent())

agent_messages = format_to_openai_function_messages(response["intermediate_steps"])
staged_messages = []
staged_messages.extend(agent_messages)
staged_messages.append(AIMessage(content=response["output"]))
memory.add(staged_messages)
print(memory.memory())

And when it executes a Playwright browser-related command, it just freezes and does nothing.

Here is an example that runs a Playwright browser-related command:

> Summarize today's financial news from Google Finance.
[SystemMessage(content='You are a helpful assistant.'), HumanMessage(content="Summarize today's financial news from Google Finance.")]

> Entering new AgentExecutor chain...

Invoking: `navigate_browser` with `{'url': 'https://www.google.com/finance'}`

But when it utilizes any other tool, such as Tavily Search, it works successfully:

> What is the weather supposed to be like in Miami tomorrow?            
[SystemMessage(content='You are a helpful assistant.'), HumanMessage(content='What is the weather supposed to be like in Miami tomorrow?')]

> Entering new AgentExecutor chain...

Invoking: `tavily_search_results_json` with `{'query': 'Miami weather forecast for tomorrow'}`

[{'url': 'https://www.weather.gov/mfl/\xa0', 'content': 'Read More > Privacy Policy Miami - South Florida Weather Forecast Office NWS Forecast Of
fice Miami - South Florida  Aviation Weather International Weather RADAR IMAGERY National Miami Radar Key West Radar Across Florida CLIMATE  Miami - South Florida11691 SW 17th StreetMiami, FL 33165305-229-4522Comments? Questions? Please Contact Us.  Last Map Update: Fri, Jan. 5, 2024 at 3:54:05 pm EST Text Product Selector: CURRENT HAZARDS OutlooksMiami - South Florida. Weather Forecast Office. NWS Forecast Office Miami - South Florida. Weather.gov > Miami ... Fri, Jan. 5, 2024 at 2:20:05 am EST. Watches, Warnings & Advisories. Zoom Out: Gale Warning: Small Craft Advisory: Rip Current Statement: ... National Weather Service Miami - South Florida 11691 SW 17th Street Miami, FL 33165 305 ...'}]The weather forecast for Miami tomorrow can be found on the National Weather Service's website for the Miami - South Florida region. You can visit [this link](https://www.weather.gov/mfl/) for the most up-to-date information on the weather forecast, including any watches, warnings, or advisories that may be in effect.                                                                                                                                            
> Finished chain.
[SystemMessage(content='You are a helpful assistant.'), HumanMessage(content='What is the weather supposed to be like in Miami tomorrow?'), AIMes
sage(content='', additional_kwargs={'function_call': {'arguments': '{"query":"Miami weather forecast for tomorrow"}', 'name': 'tavily_search_results_json'}}), FunctionMessage(content='[{"url": "https://www.weather.gov/mfl/\xa0", "content": "Read More > Privacy Policy Miami - South Florida Weather Forecast Office NWS Forecast Office Miami - South Florida  Aviation Weather International Weather RADAR IMAGERY National Miami Radar Key West Radar Across Florida CLIMATE  Miami - South Florida11691 SW 17th StreetMiami, FL 33165305-229-4522Comments? Questions? Please Contact Us.  Last Map Update: Fri, Jan. 5, 2024 at 3:54:05 pm EST Text Product Selector: CURRENT HAZARDS OutlooksMiami - South Florida. Weather Forecast Office. NWS Forecast Office Miami - South Florida. Weather.gov > Miami ... Fri, Jan. 5, 2024 at 2:20:05 am EST. Watches, Warnings & Advisories. Zoom Out: Gale Warning: Small Craft Advisory: Rip Current Statement: ... National Weather Service Miami - South Florida 11691 SW 17th Street Miami, FL 33165 305 ..."}]', name='tavily_search_results_json'), AIMessage(content="The weather forecast for Miami tomorrow can be found on the National Weather Service's website for the Miami - South Florida region. You can visit [this link](https://www.weather.gov/mfl/) for the most up-to-date information on the weather forecast, including any watches, warnings, or advisories that may be in effect.")]

Expected behavior

The behavior of the Playwright Browser should return results to an input in a relatively prompt manner, but it currently freezes and returns.

dosubot[bot] commented 6 months ago

🤖

Hello @CurtMRosenbladWheeler! I'm Dosu, an AI bot here to assist you with resolving issues, answering questions, and helping you contribute to LangChain. While we're waiting for a human maintainer to join us, feel free to ask me anything you might need help with. Let's work together to make LangChain even better!

Thank you for providing such a detailed description of the issue you're facing. It seems like the Playwright Browser is not behaving as expected within the LangChain framework. Let's try to troubleshoot this issue.

Firstly, it's important to ensure that the Playwright Python library is correctly installed and configured, as it is a crucial dependency for the Playwright Browser to function correctly within the LangChain framework. You can verify this by running the following command in your terminal:

pip show playwright

This command should return information about the installed Playwright package. If it doesn't return anything, it means that the Playwright library is not installed, and you should install it using the following command:

pip install playwright

After installing the Playwright library, you need to run the following command to download the necessary browser binaries:

playwright install

If the Playwright library is correctly installed and configured but the issue persists, it might be related to the specific implementation of the Playwright tools within the LangChain framework.

From the code you've shared, it seems like the issue arises when the navigate_browser tool is invoked. This tool is likely used to navigate to different URLs or pages within the browser. If the browser is freezing during this operation, it might be due to network issues, or the page taking too long to load.

One way to debug this issue is to run the Playwright Browser in non-headless mode by setting the headless parameter to False when creating the browser:

playwright_async_browser = create_async_playwright_browser(headless=False)

Running the browser in non-headless mode will open a graphical browser window, allowing you to see what's happening when the navigate_browser tool is invoked.

If the issue still persists, it would be helpful to have more information about the error. Are there any error messages being displayed in the console when the browser freezes? This information could provide valuable clues about what's causing the issue.

I hope this helps! Please let me know if you have any other questions or if there's anything else I can do to assist you.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

thiswillbeyourgithub commented 5 months ago

I had the same issue in a gradio app. I moved the use of function create_sync_playwright_browser to a piece of code that was running at startup instead of in a function called by gradio and it worked.

Edit: not sure it's a good fix actually because I now get greenlet.error: cannot switch to a different thread and struggle to find a workaround (I'm using the sync playwright version and don't plan on converting my codebase to async)

thiswillbeyourgithub commented 5 months ago

In the end I moved to async and am getting the same issue :

[tool/start] [1:chain:AgentExecutor > 4:tool:navigate_browser] Entering Tool run with input:
"{'url': 'https://duckduckgo.com/?q=sam+altman+birthdate'}"

hangs indefinitely

mantrakp04 commented 3 months ago

Facing a similar issue with FastAPI, cannot use either sync or async, tried messing around with all possible combos but nothing seems to work

eav-solution commented 1 month ago

Me too @@

langchain-ai / langchain