search

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.14k stars 1.88k forks source link

feat: account for other sitemap names #133

Closed Skeyelab closed 5 months ago

Skeyelab commented 5 months ago

I ran across a site that had a sitemap i wanted to crawl named sitemap_recent.xml

This PR changes the check for isUrlASitemap to the following:

const isUrlASitemap = /sitemap.*\.xml$/.test(config.url);

This code will set isUrlASitemap to true if config.url ends with 'sitemap' followed by any characters and then '.xml'. For example, it will match sitemap123.xml, sitemap-abc.xml, sitemap.xml, etc.

github-actions[bot] commented 5 months ago

:tada: This PR is included in version 1.3.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: