Improvement: Improve URL Scraping/Ingestion

[rmusser01]

Issue to track improvements/ideas for URL Scraping & Ingestion

• [ ] Add custom cookie support
• [ ] Instructions for adding custom browser-addons to the scraping browser
• [ ] Support for identifying article Title/Author name(s)
• [ ] Support for identifying article publish date

Seems like I can possibly skip all this if I use: https://github.com/ArchiveBox/ArchiveBox/wiki + https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles And then use it via the cli to archive the page, and then extract the text using Trafilatura and do modifications on the data from there.

Browser Plugin

• https://github.com/deathau/markdownload

Ingestion

• https://trafilatura.readthedocs.io/en/latest/troubleshooting.html#beyond-raw-html

Replicating structure using markdown https://github.com/AnswerDotAI/web2md/tree/main

Scraping:

• https://python.langchain.com/v0.1/docs/integrations/document_loaders/web_base/ Firecrawl:
  • https://docs.firecrawl.dev/introduction
  • https://github.com/mendableai/firecrawl Scrapper
• https://github.com/amerkurev/scrapper
• https://github.com/mishushakov/llm-scraper
• https://github.com/romansky/dom-to-semantic-markdown Headless Browsers
• https://github.com/ultrafunkamsterdam/nodriver
• https://github.com/dhamaniasad/HeadlessBrowsers
• https://github.com/iamadamdev/bypass-paywalls-chrome
• https://github.com/daijro/browserforge

Spoofing client

• https://github.com/yifeikong/curl_cffi
• https://scrapfly.io/web-scraping-tools/curl-python/curl_cffi

Storage: https://github.com/iansinnott/full-text-tabs-forever

async https://jacobpadilla.com/articles/recreating-asyncio https://aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html

Site-Specific stuff

• Reddit: https://github.com/JosefAlbers/rd2md

[rmusser01]

Add hashing of ingested article content to identify changes made between scrapes

[rmusser01]

Modification of headless browser for injection of cookies/user:pass/plugins

[rmusser01]

https://github.com/muchdogesec/history4feed

[rmusser01]

Confluence scraping: https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/document_loaders/confluence.py

[rmusser01]

HTML https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/document_loaders/html.py

[rmusser01]

Gitbook: https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/document_loaders/gitbook.py

[rmusser01]

txt files stored in a git repo: https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/document_loaders/git.py

[rmusser01]

https://github.com/TheBlewish/Web-LLM-Assistant-Llama-cpp

[rmusser01]

https://github.com/paul-gauthier/aider/blob/main/aider/scrape.py