search

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.16k stars 1.88k forks source link

craw openai --get nothing #55

Open zyxcambridge opened 7 months ago

zyxcambridge commented 7 months ago

import { Config } from "./src/config";

export const defaultConfig: Config = { url: "https://openai.com/", match: "gpt", maxPagesToCrawl: 100, outputFileName: "outputzyx.json", };

INFO PlaywrightCrawler: Starting the crawler. INFO PlaywrightCrawler: Crawling: Page 1 / 100 - URL: https://openai.com/... INFO PlaywrightCrawler: All requests from the queue have been processed, the crawler will shut down. INFO PlaywrightCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":6738,"requestsFinishedPerMinute":9,"requestsFailedPerMinute":0,"requestTotalDurationMillis":6738,"requestsTotal":1,"crawlerRuntimeMillis":6867} INFO PlaywrightCrawler: Finished! Total 1 requests: 1 succeeded, 0 failed. {"terminal":true}

mvavassori commented 7 months ago

I get the same error with crawling Next.js docs.