search

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.17k stars 1.89k forks source link

Any way to use a sitemap.xml for the crawler? #16

Closed spawn9859 closed 7 months ago

spawn9859 commented 7 months ago

I cant seem to get the crawler to crawl every extension of a website, it sometimes misses a lot. The site does have a sitemap.xml with every link I would want though, is there any way to use that? If so, then how?

guillermoscript commented 7 months ago

hey, I added sitemap support in my pr, i also needed sitemap support

github-actions[bot] commented 7 months ago

:tada: This issue has been resolved in version 1.0.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: