One consistent feedback we get around PQ and IVF is that there is limited visibility into the failure cases. Part of this is because the errors are thrown on the faiss side and we dont return stack traces in Rest response.
This makes it difficult to use PQ and IVF.
To make it better, we should explicitly detect and message the following cases:
[ ] For PQ, explicitly check in OpenSearch an invalid configuration where m does not divide dimension
[ ] For PQ/IVF, check the number of training points matches the minimum clustering criteria defined in faiss
[ ] If there is not enough memory, explicitly say that there is not enough memory.
Checking these 3 configurations will cover 90% of the training failures that occur. With the changes made in disk-based, it should be relatively easy to do this.
Description
One consistent feedback we get around PQ and IVF is that there is limited visibility into the failure cases. Part of this is because the errors are thrown on the faiss side and we dont return stack traces in Rest response.
This makes it difficult to use PQ and IVF.
To make it better, we should explicitly detect and message the following cases: [ ] For PQ, explicitly check in OpenSearch an invalid configuration where m does not divide dimension [ ] For PQ/IVF, check the number of training points matches the minimum clustering criteria defined in faiss [ ] If there is not enough memory, explicitly say that there is not enough memory.
Checking these 3 configurations will cover 90% of the training failures that occur. With the changes made in disk-based, it should be relatively easy to do this.
Related Issues
1332
593