search

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.58k stars 1.97k forks source link

How to crawl Single Page Application(SPA) #149

Open ouyh1111 opened 7 months ago

ouyh1111 commented 7 months ago

Such as: https://music.163.com/ ,The site is a spa; pages url all like that: https://music.163.com/#/my/, the config.ts is below,but the crawler only crawl the url(https://music.163.com/) and save to output.json

config.ts `import { Config } from "./src/config";

export const defaultConfig: Config = { url: "https://music.163.com/", match: "https://music.163.com/#/**", maxPagesToCrawl: 1000, outputFileName: "output.json", maxTokens: 2000000, };`