BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.16k stars 1.88k forks source link

Added Directory Exclusion #27

Open bleachedsleet opened 7 months ago

bleachedsleet commented 7 months ago

Recursive directory exclusion (#9) and user configurable through config.ts. A list of excluded parent directories is read into an array to be checked against during the crawling process and skipped if found. Pretty quick and dirty and could probably use some optimization.

Tidied up some related comments as well to make things a little clearer.

vaibhavkumar-sf commented 7 months ago

Preparing review...