michelle123lam
commented
4 months ago Hi, thanks for trying out LLooM! Yes, this seems related to the same issue that prompted this prior pull request (and that came up in this external OpenAI community thread). Basically, there are issues with the OpenAI embeddings API when there are too many embeddings requested at once, which is probably why it only came up when you tested with the much larger dataset. These issues should be fixed when I update the production package to reflect the current state of the repo (which has the PR merged in). I'll go ahead and do that and follow up in the thread when that's been updated!
andreawwenyi
Hi! Thanks for developing LLooM! I am eager to try it out on my dataset of 1000 documents, but I kept getting this error at the "Cluster" stage:
BadRequestError: Error code: 400 - {'error': {'message': "'$.input' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference.", 'type': 'invalid_request_error', 'param': None, 'code': None}}. It seems like it's related to the get_embedding function. I am using python==3.11, openai==1.33.0, default embedding model specified in LLooM. I would not get this error if I am only playing with 200 documents, but once I increase the document size I would get the error. Do you might know what's causing the issue and what could be a workaround? Thank you!