some html like https://www.abc.com/file/abc.pdf couldn't be crawled

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

https://www.builder.io/blog/custom-gpt

ISC License

18.15k stars 1.88k forks source link

some html like https://www.abc.com/file/abc.pdf couldn't be crawled #115

Open cuterv opened 6 months ago

cuterv commented 6 months ago

WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. page.goto: net::ERR_ABORTED at https://www.abc.com/file/abc.pdf Call log:

navigating to "https://www.abc.com/file/abc.pdf", waiting until "load"

{"id":"abcdef","url":https://www.abc.com/file/abc.pdf","retryCount":1}