search

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.15k stars 1.88k forks source link

maxpage did not work well #110

Open bcharityfi opened 6 months ago

bcharityfi commented 6 months ago

I set maxpage as 100, 300, or 1000, but it always stopped at 56.

was there anyone to encounter the same issue with me?

And any solution you may have?

PipeDream941 commented 6 months ago

Hello, I encountered the same situation, but I don't think this is a bug. It might be a feature that needs to be added in the future.

This is because the gpt-crawler can only crawl static pages, and it cannot automatically crawl pages that require interaction with dynamic content. Therefore, it won't reach the set maxpage.