provider: SambaNova FastAPI implementation.

[ilsubyeega]

This PR implements SambaNova FastAPI implementation. Which is currently private-beta, but the request codes are in sambanova/ai-starter-kit repository. Whole codebases are copied from /src/provider/groq directory.

Related #407. But this does not implement any other apis that are mentioned.

It's working when quick and simple test, but I must verify that it works properly, so I've submitted this as Draft.

[ilsubyeega]

I think this is okay for now.

[VisargD]

Hey! Thanks for the PR. We recently released a openai base provider which can be used for all openai compatible providers. It would be great if you can use it for this provider as well because its OpenAI compliant. You can check the Cerebras integration as a reference on how to use the base provider.

[ilsubyeega]

We recently released a openai base provider which can be used for all openai compatible providers.

@VisargD I'm curious that this open-ai-base provides stream-chatComplete. I can only see that transform non-stream repsonse but not stream one (and even it has custom property at sambanova).[1]

Also, i doubt at https://github.com/Portkey-AI/gateway/blob/1a4ab70fbbaea572c85f075beb5e3dd370d99065/src/providers/open-ai-base/index.ts#L32

chatCompleteParams#defaultValues?: Record<string, string> looks wrongly designed. PrarameterConfig.default? provides any type, but #defaultValues only appliable with string but not boolean and numbers. I'm just putting string at this moment, but not ideal i guess.

[1]: Edit; seems groq got reverted, nevermind.

[b4s36t4]

@ilsubyeega I have tried running the code locally but keep on getting 401, were you able to test this locally if so can you share a video recording of the response.

Following is the curl I have tried.

curl --request POST \
  --url http://localhost:8787/v1/chat/completions \
  --header 'authorization: key' \
  --header 'content-type: application/json' \
  --header 'x-portkey-provider: sambanova' \
  --data '{
  "messages": [
    {
      "role": "user",
      "content": "Hello, world"
    }
  ]
}'

[ilsubyeega]

were you able to test this locally if so can you share a video recording of the response.

@b4s36t4 Here you go.

Screencast from 2024-09-06 20-54-31.webm

[b4s36t4]

@ilsubyeega Sorry for the delay. But I just have got merged the typing issue for the defaluValues for the base provider. If possible please update the PR to use stream with boolean value instead of string value. Then we can merge it.

[ilsubyeega]

Here are some concerns: Default parameter does not affect isStreamingMode, which requires stream paramter on request even though it should appended as default.

https://github.com/Portkey-AI/gateway/blob/7eaf93235cd81d66e969628bba55e56762186a6d/src/handlers/handlerUtils.ts#L459-L491 transformToProviderRequest parses with default provider config, if it done correctly, but it does not affect isStreamingMode, make it fails.

Since stream is not required param, so it does not append stream: true on body. https://github.com/Portkey-AI/gateway/blob/7eaf93235cd81d66e969628bba55e56762186a6d/src/services/transformToProviderRequest.ts#L116-L128

Here is possible workarounds:

1. /src/handlers/handlerUtils.ts#tryPost, it should be reverify after transformToProviderRequest, e.g

   if ((transformedRequestBody as { stream?: boolean })['stream'] == true) isStreamingMode = true;

2. Two options: add require params on /src/provider/open-ai-base/index.ts#chatCompleteParams, or remove paramConfig.required check to /src/services/transformToProviderRequest.ts#transformToProviderRequestJSON

   stream: {
     param: 'stream',
     ...(defaultValues?.stream && { default: defaultValues.stream, required: true }), // <--
   },

     else if (
       paramConfig &&
       // paramConfig.required && <--- remove this
       paramConfig.default !== undefined
     ) {

   it seems default option is useless when required=false, so the last option looks ideal for me.

Since the provider only supports streaming-mode at this moment, which is unusual, so this issue being kinda tricky. Or we can just ignore this since the client(requesting to gateway) should set stream=true explicitly, otherwise it will break parsing responses.

[b4s36t4]

@ilsubyeega Sorry, I didn't understand what you're trying to say. If possible can we take this over discord chat. b4s36t4 is my discord hit me up.

[b4s36t4]

Also sambanova does support non-stream request as well.

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "content": "It seems like you've mentioned a name, \"Mahesh\". Would you like to know more about someone with this name, or perhaps discuss something related to it?"
      }
    }
  ],
  "created": 1725703777,
  "id": "08f08fe8-889a-4cdb-b406-e4f8077b2537",
  "model": "Meta-Llama-3.1-8B-Instruct",
  "object": "chat.completion",
  "system_fingerprint": "fastcoe",
  "usage": {
    "completion_tokens": 33,
    "completion_tokens_after_first_per_sec": 483.3644057419852,
    "completion_tokens_after_first_per_sec_first_ten": 889.0098453777595,
    "completion_tokens_per_sec": 355.31535860884566,
    "end_time": 1725703777.6294942,
    "is_last_response": true,
    "prompt_tokens": 11,
    "start_time": 1725703777.536619,
    "time_to_first_token": 0.0266726016998291,
    "total_latency": 0.09287524223327637,
    "total_tokens": 44,
    "total_tokens_per_sec": 473.7538114784609
  }
}

[ilsubyeega]

Uh... While testing i got nonstream response so I thought that the gateway caching it, but was sambanova side, it works sometimes and sometimes not.

[vrushankportkey]

Hey @ilsubyeega - can you please share the email/name of person from Sambanova you were in touch with? We can reach out to to inform about this integration and discuss what more can be done.

[ilsubyeega]

Hey @ilsubyeega - can you please share the email/name of person from Sambanova you were in touch with? We can reach out to to inform about this integration and discuss what more can be done.

I am just a private beta test api user and have no connection with sambanova. I received the mail through salesforce.