How to process webpage content exceeding the GPT length limit

EZ-hwh / AutoScraper

Official implement of paper "AutoCrawler: A Progressive Understanding Web Agent for Web Crawler Generation"

Apache License 2.0

403 stars 34 forks source link

Closed yaoxiaona closed 1 month ago

yaoxiaona commented 4 months ago

The length of webpage content will exceed the length limit of ChatGPT, right? How will the excess be handled?

EZ-hwh commented 3 months ago

Yes, we recently updated a new vision of code that can handle the long HTML that exceeds the limit of LLM.