search

phidatahq / phidata

Build AI Agents with memory, knowledge, tools and reasoning. Chat with them using a beautiful Agent UI.
https://docs.phidata.com
Mozilla Public License 2.0
15.58k stars 2.14k forks source link

Update Firecrawl tool with `Async Crawling` and `Map` functionality. #1349

Open AbhishekRP2002 opened 1 month ago

AbhishekRP2002 commented 1 month ago

current implementation of firecrawl tool within phidata includes scrape_url and crawl_url methods. However, there are few other new methods / features introduced in the original python SDK like async crawling and map method. i would like to include them in the phidata firecrawl tool since I believe these updates will be valuable to users.

To understand more about the features, one can refer this: https://docs.firecrawl.dev/sdks/python

pls lemme know how we take this forward or if there's any clarification required at my end.

ccing: @manthanguptaa for speeding up the process.

manthanguptaa commented 1 month ago

Hey @AbhishekRP2002, thanks for showing interest in contributing to Phidata. This sounds like a wonderful addition to the firecrawl tool. Would you also like to add checking crawl status, cancelling a crawl, and crawling a website with websockets as well?

AbhishekRP2002 commented 1 month ago

yes absolutely. thanks !

manthanguptaa commented 1 month ago

Perfect! I look forward to your PR

AbhishekRP2002 commented 4 weeks ago

hi @manthanguptaa quick query: does phidata allow using tools / defining tools explicitly which are already part of existing pre-defined toolkits ? For eg.

agent = Agent(tools=[scrape_website function from FirecrawlToools, search_exa function from ExaTools...], markdown = True, reasoning = True, show_tool_calls=True,)
AbhishekRP2002 commented 4 weeks ago

hi @manthanguptaa I have created a PR, however I made a mistake, I made some changes with my org account due to which I couldn't the push the changes to my branch. as an impact to this I had filter-branch and add my personal creds, leading to a series of all commits made in past.

apologies again. i will keep this in mind. pls lemme know if there are any feedbacks or suggestions.

AbhishekRP2002 commented 2 weeks ago

hi @manthanguptaa pls lemme know if there's any other review / requested changes that needs to be addressed. will be glad to take them up !

manthanguptaa commented 1 week ago

Hey @AbhishekRP2002, give me some time to review the PR. I will try to review them on the weekend

AbhishekRP2002 commented 1 week ago

Perfect. I have closed the current pr due to long chain of previous commits traced from og commits of the repo. I'll create a fresh PR post re-forking and will link to the issue number by Friday. This should give you some time to review the code over the weekend Cc: @manthanguptaa