This PR introduces support for various OpenAI embedding models, implementing response length variability based on a configuration file named openai_deployment_embedding_config.json. This enhancement allows for flexible and customizable embedding model usage within the application.
Key Changes
Configuration-Based Model Selection:
The PR adds support for selecting different embedding models based on a configuration file (openai_deployment_embedding_config.json).
If the configuration file is not found, the default embedding model used is text-embedding-ada-002 with an embedding size of 1536.
Support for Additional Models:
The PR adds support for the text-embedding-3 and subsequent models.
Corresponding tests have been updated to cover these new models, ensuring robustness and reliability.
Detailed Implementation
Configuration File: The openai_deployment_embedding_config.json file contains model-specific settings that determine the embedding length and model name embedding models. An example can be seen below.
In the absence of the configuration file, the application defaults to using the text-embedding-ada-002 model with a embedding size of 1536.This ensures backward compatibility and provides a seamless experience for users who may not have the configuration file set up.
Testing and Validation:
The test suite has been updated to include tests for the text-embedding-3 and later models.
These tests ensure that the new models are correctly supported and integrated within the application's functionality.
PR for OpenAI Embedding Models Support
Overview
This PR introduces support for various OpenAI embedding models, implementing response length variability based on a configuration file named openai_deployment_embedding_config.json. This enhancement allows for flexible and customizable embedding model usage within the application.
Key Changes
Configuration-Based Model Selection:
Support for Additional Models:
Detailed Implementation
In the absence of the configuration file, the application defaults to using the text-embedding-ada-002 model with a embedding size of 1536.This ensures backward compatibility and provides a seamless experience for users who may not have the configuration file set up.
Testing and Validation:
The test suite has been updated to include tests for the text-embedding-3 and later models. These tests ensure that the new models are correctly supported and integrated within the application's functionality.