Web Sources's "website link mode" does not scrape recursively the entire web site

neo4j-labs / llm-graph-builder

Neo4j graph construction from unstructured data using LLMs

https://neo4j.com/labs/genai-ecosystem/llm-graph-builder/

Apache License 2.0

2.07k stars 310 forks source link

Web Sources's "website link mode" does not scrape recursively the entire web site #655

Open marcofiocco opened 1 month ago

marcofiocco commented 1 month ago

It would be a very useful feature.

If you cannot implement it, I can imlement my own web scraper, but what would be the best way to load all the scraped webpages? Even Browse mode does not allow to specify whole folders, but just multi-selection files

jexp commented 3 weeks ago

I think as this is a mass processing job, it would make sense to use the underlying python code with LLMGraphTransformer in Langchain.

https://python.langchain.com/v0.1/docs/use_cases/graph/constructing/#llm-graph-transformer