PR: Update generator support for embeddings (vary with model and support dimension request parameter)

[MDUYN]

PR for OpenAI Embedding Models Support

Overview

This PR introduces support for various OpenAI embedding models, implementing response length variability based on a configuration file named openai_deployment_embedding_config.json. This enhancement allows for flexible and customizable embedding model usage within the application.

Key Changes

Configuration-Based Model Selection:

• The PR adds support for selecting different embedding models based on a configuration file (openai_deployment_embedding_config.json).
• If the configuration file is not found, the default embedding model used is text-embedding-ada-002 with an embedding size of 1536.

Support for Additional Models:

• The PR adds support for the text-embedding-3 and subsequent models. Corresponding tests have been updated to cover these new models, ensuring robustness and reliability.

  Detailed Implementation

• Configuration File: The openai_deployment_embedding_config.json file contains model-specific settings that determine the embedding length and model name embedding models. An example can be seen below.

  {
  "text-embedding-3-small": {
      "model": "text-embedding-3-small",
      "size": "512"
  },
  "text-embedding-3-medium": {
      "model": "text-embedding-3-medium",
      "size": "768"
  },
  "text-embedding-3-large": {
      "model": "text-embedding-3-large",
      "size": "1024"
  },
  "text-embedding-3-xlarge": {
      "model": "text-embedding-3-xlarge",
      "size": "1536"
  },
  "text-embedding-ada-001": {
      "model": "text-embedding-ada-001",
      "size": "768"
  },
  "text-embedding-ada-002": {
      "model": "text-embedding-ada-002",
      "size": "1536"
  }
  }

In the absence of the configuration file, the application defaults to using the text-embedding-ada-002 model with a embedding size of 1536.This ensures backward compatibility and provides a seamless experience for users who may not have the configuration file set up.

Testing and Validation:

The test suite has been updated to include tests for the text-embedding-3 and later models. These tests ensure that the new models are correctly supported and integrated within the application's functionality.