How to limit the hierarchy of pages to be crawled?

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

https://www.builder.io/blog/custom-gpt

ISC License

18.15k stars 1.88k forks source link

Open wywywy1990 opened 6 months ago

wywywy1990 commented 6 months ago

How to limit the hierarchy of pages to be crawled? For example, restricting the entry of links on this page to only sniff links within two levels