Issue: how to fetch sub url using langchain

langchain-ai / langchain

🦜🔗 Build context-aware reasoning applications

MIT License

94.91k stars 15.37k forks source link

Issue you'd like to raise.

below is my code with sync_playwright() as p: browser = p.chromium.launch() navigate_tool = NavigateTool(sync_browser=browser) extract_hyperlinks_tool = ExtractHyperlinksTool(sync_browser=browser) for url in urls: print(url,"url is ----------------------") navigate_tool._arun(url) print(navigate_tool._arun(url)) hyperlinks = extract_hyperlinks_tool._arun() for link in hyperlinks: print(link,"link is ------------------------------------------")

and i am getting these error <coroutine object NavigateTool._arun at 0x7f0ab738f0c0> /home/hs/CustomBot/accounts/common_langcain_qa.py:122: RuntimeWarning: coroutine 'NavigateTool._arun' was never awaited print(navigate_tool._arun(url)) RuntimeWarning: Enable tracemalloc to get the object allocation traceback Internal Server Error: /create-project/ Traceback (most recent call last): File "/home/hs/env/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner response = get_response(request) File "/home/hs/env/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, callback_kwargs) File "/home/hs/env/lib/python3.8/site-packages/django/views/generic/base.py", line 69, in view return self.dispatch(request, *args, *kwargs) File "/home/hs/env/lib/python3.8/site-packages/django/views/generic/base.py", line 101, in dispatch return handler(request, args, kwargs) File "/home/hs/CustomBot/user_projects/views.py", line 1776, in post file_crawl_status, file_index_status = generate_embeddings( File "/home/hs/CustomBot/accounts/common_langcain_qa.py", line 124, in generate_embeddings for link in hyperlinks:

TypeError: 'coroutine' object is not iterable [16/Dec/2023 15:59:50] "POST /create-project/ HTTP/1.1" 500 89444 /usr/lib/python3.8/pathlib.py:755: RuntimeWarning: coroutine 'ExtractHyperlinksTool._arun' was never awaited return self._cached_cparts RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Suggestion:

No response

🤖

Hi there @deepak-habilelabs! Great to see you again, I hope you're pushing pixels and cracking codes as usual. Let's take a look at your issue.

The error you're encountering, 'coroutine' object is not iterable, is likely due to the fact that you're trying to iterate over a coroutine object, which is not allowed in Python. Coroutines are meant to be awaited, not iterated over.

In your code, you're calling the _arun method on navigate_tool and extract_hyperlinks_tool but you're not awaiting these calls. The _arun method is a coroutine and therefore needs to be awaited. Here's how you can modify your code:

with sync_playwright() as p:
    browser = p.chromium.launch()
    navigate_tool = NavigateTool(sync_browser=browser)
    extract_hyperlinks_tool = ExtractHyperlinksTool(sync_browser=browser)
    for url in urls:
        print(url,"url is ----------------------")
        await navigate_tool._arun(url)
        print(await navigate_tool._arun(url))
        hyperlinks = await extract_hyperlinks_tool._arun()
        for link in hyperlinks:
            print(link,"link is ------------------------------------------")

In this modified code, await navigate_tool._arun(url) and await extract_hyperlinks_tool._arun() correctly await the coroutines and gather their results.

Please note that you need to run this code in an asynchronous context because you're using await. If you're not already doing so, you can define an asynchronous function and run it using an event loop. Here's an example:

import asyncio

async def main():
    # Your code here

asyncio.run(main())

I hope this helps! If you have any other questions, feel free to ask.

For more information about coroutines and asynchronous programming in Python, you can refer to the Python documentation.

For more information about the NavigateTool._arun method and the ExtractHyperlinksTool._arun method in the LangChain framework, you can refer to the LangChain repository.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

langchain-ai / langchain