Long input document - Githubissues

GDD-Nantes / LLM4SchemaOrg

0 stars 1 forks source link

Closed mhoangvslev closed 8 months ago

mhoangvslev commented 9 months ago

For input webpage that is too long, typically from database-dump websites, OpenAI rejects to process requests, even with MapReduce process.
The markup is too long (for example with 234ece1a0169bec40c5f453b294e808b_Book, there are +1k lines) and the model will stop writting output.

mhoangvslev commented 8 months ago

Use chunkipy