Tracking: Web Scraping Ingestion Pipeline

[rmusser01]

Issue is to track efforts to improve the web scraping pipeline.

• [ ] Implement Pycookie
• [ ] Implement checks for custom scraper integration (if URL matches a predefined list, use the scraper for that URL/technology)
• [ ] Confluence
• [ ] Reddit - https://github.com/JosefAlbers/rd2md
• [ ] Fully fix + Improve https://github.com/rmusser01/dom-to-semantic-markdown and implement it (see also https://github.com/web-arena-x/webarena/blob/main/browser_env/processors.py#L47)
• [ ] Implement form of Browser UA Spoofing
• https://github.com/yifeikong/curl_cffi
• https://scrapfly.io/web-scraping-tools/curl-python/curl_cffi
• [ ] #385

Other Scraper Implementations:

• https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/extraction_strategy.py & https://github.com/Aider-AI/aider/blob/main/aider/scrape.py
• https://github.com/amerkurev/scrapper

[rmusser01]

https://github.com/unclecode/crawl4ai

[rmusser01]

https://github.com/JohannesKaufmann/html-to-markdown