BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL
https://www.builder.io/blog/custom-gpt
ISC License
18.14k stars 1.88k forks source link

extracting text in hidden div blocks #157

Open udgithub opened 3 months ago

udgithub commented 3 months ago

Hello !

I like your gpt-crawler, but sometimes it does not return all text within a page. E.g. on the FAQ page of our computing center users find all questions but our CMS hides the answers by using style information like this: style="height: 0px; overflow: hidden; display: none;" Unfortunately your crawler does not return the answers. Is it possible or configurable to make your crawler return also hidden text?

Thanks a lot in advance for your help and time.

Best regards, Uwe