Feature: Support for crawling dynamic javascript heavy site

[indrajithi]

Description:

Enhance the existing web crawler to support crawling and extracting content from websites that rely heavily on JavaScript for rendering their content. This feature will involve integrating a headless browser to accurately render and interact with such pages.

Objectives:

• Enable the crawler to fetch and parse content from JavaScript-heavy sites.
• Use a headless browser to render JavaScript content. (explore playwright-python)
• Ensure compatibility with the existing crawler structure and options.
• Maintain the ability to switch between the default fetching method and the headless browser.

Design Considerations:

• Single Headless Browser Instance:
  • Use a single instance of a headless browser to handle multiple asynchronous requests, reducing resource consumption.
• Concurrency Management:
  • Utilize asyncio and a semaphore to manage concurrent requests within the same browser context.
  • Integrate the asynchronous fetching logic with our existing web crawler structure.
• Error Handling:
  • Ensure proper error handling and resource cleanup. (no zombie browsers, they are already headless :p)
  • Fall back to default fetching mode when there is a error with the headless browser. (keep the user informed)

[indrajithi]

Blocked by #17

[Mews]

What does blocked mean? I'd like to work on this but do you think I shouldn't?

[indrajithi]

@Mews You can work on this. I want to complete #17 before picking this up. I have merged the MR for that. Although there are a few more things to be done for #17. I believe this issue can be unblocked.

Since this is going to be relatively bigger story. Let us first discuss the approach and spec out the requirements and acceptance criteria.

[Mews]

Alright makes sense I'll work on the v1 milestone too then. I'll pick this up when v1 gets released 👍

[indrajithi]

You can create Issues for things in #24 you find interesting and pick it up. Meanwhile I will spec out some details in this Issue. Also I think we should have this in v1.