Added document chunking

a16z-infra / ai-getting-started

A Javascript AI getting started stack for weekend projects, including image/text models, vector stores, auth, and deployment configs

https://ai-getting-started.com/

MIT License

4k stars 643 forks source link

Closed fmhall closed 1 year ago

fmhall commented 1 year ago

When uploading a corpus of documents with more than 8191 tokens, the Pinecone API returns an error.

This PR chunks the documents and sends them in a list.