This pull request implements the support for multiple embedding models. It enables customers to set up multiple embedding models as supported by AutoModels in a single deployment. Each embedding model is backed by a separate table in the PostgreSQL database. A new field named 'modelRefKey' is added to the embedding model entity to uniquely identify the embedding models.
Changes in the CLI:
Deploy Flow: Customers will need to specify how many embedding models they want to support. Within each model, they need to specify the modelRefKey for each model.
Document Upload Flow: Customers also need to specify the modelRefKey to indicate which embedding model to use.
Indexing Pipeline:
The embedding model can now be specified from the execution input.
Corpus APIs:
The modelRefKey is added to indicate which embedding model to use.
Web UI:
A new dropdown list named 'Embedding Models' is added to the 'Semantic Search' section in the chat settings, allowing customers to select the embedding model for chats.
The developer tools 'Corpus Search' and 'Embeddings' are updated to support embedding model selection.
Does this PR introduce a breaking change?
N/A
Related Issues/Discussion
N/A
How Has This Been Tested?
The PR has been tested on my own test environment.
What environment was this tested on?
N/A
Screenshots (if appropriate)
PR Checklist
[ ] Have you added/updated documentation? - YES.
[ ] Have you written new tests for your core changes, as applicable? - N/A
[ ] Have you successfully ran build and tests with your changes locally? - Yes
IMPORTANT: Please review the CONTRIBUTING.md file for detailed contributing guidelines.
Description
This pull request implements the support for multiple embedding models. It enables customers to set up multiple embedding models as supported by AutoModels in a single deployment. Each embedding model is backed by a separate table in the PostgreSQL database. A new field named 'modelRefKey' is added to the embedding model entity to uniquely identify the embedding models.
Deploy Flow: Customers will need to specify how many embedding models they want to support. Within each model, they need to specify the modelRefKey for each model.
Document Upload Flow: Customers also need to specify the modelRefKey to indicate which embedding model to use.
Indexing Pipeline: The embedding model can now be specified from the execution input.
Corpus APIs: The modelRefKey is added to indicate which embedding model to use.
Web UI: A new dropdown list named 'Embedding Models' is added to the 'Semantic Search' section in the chat settings, allowing customers to select the embedding model for chats. The developer tools 'Corpus Search' and 'Embeddings' are updated to support embedding model selection.
Does this PR introduce a breaking change?
N/A
Related Issues/Discussion
N/A
How Has This Been Tested?
The PR has been tested on my own test environment.
Screenshots (if appropriate)
PR Checklist
IMPORTANT: Please review the CONTRIBUTING.md file for detailed contributing guidelines.