epam / ai-dial-core

The main component of AI DIAL, which provides unified API to different chat completion and embedding models, assistants, and applications
https://epam-rail.com
Apache License 2.0
339 stars 18 forks source link

feat: allow to pass extra data to upstreams endpoints #470

Closed alekseyvdovenko closed 1 month ago

alekseyvdovenko commented 1 month ago

Occasionally for providing a sophisticated load balancing it's essential to pass an extra metadata to the upstreams, so the destination/adapter will have a better context of how to process the request. The idea is to extend the current configuration for the upstreams by introducing a new optional block extraData which will be passed to a particular upstream endpoint as header X-UPSTREAM-EXTRA-DATA.

Example of the configuration:

{
  "models": {
    "llm": {
      "type": "chat",
      "upstreams": [
        {
          "extraData": {
            "<key-1>":"<value-1>",
            "<key-2>":"<value-2>"
          }
        },
        {
          "extraData": {
            "<key-3>":"<value-3>",
            "<key-4>":"<value-4>"
          }
        }
      ]
    }
  }
}
alekseyvdovenko commented 1 month ago

/deploy-review

Environment URL: https://chat-ai-dial-core-pr-470.nightly-test.deltixhub.io E2E tests status: success

alekseyvdovenko commented 1 month ago

This is the blocker for https://github.com/epam/ai-dial-adapter-bedrock/issues/126

sdryapko commented 1 month ago

@astsiapanay please review

alekseyvdovenko commented 1 month ago

/deploy-review

Environment URL: https://chat-ai-dial-core-pr-470.nightly-test.deltixhub.io E2E tests status: failed

alekseyvdovenko commented 1 month ago

/deploy-review

Environment URL: https://chat-ai-dial-core-pr-470.nightly-test.deltixhub.io E2E tests status: success