Split embeddings into batched requests in llm.py

michelle123lam / lloom

Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM (CHI 2024 paper). LLooM automatically surfaces high-level concepts to analyze unstructured text.

https://stanfordhci.github.io/lloom

BSD 3-Clause "New" or "Revised" License

45 stars 12 forks source link

Split embeddings into batched requests in llm.py #8

Closed justinebreuch closed 3 weeks ago

justinebreuch commented 1 month ago

For many documents, you hit limits on text length if you don't split them into multiple requests

https://platform.openai.com/docs/guides/embeddings/embedding-models