Feat: Provider Tests - Githubissues

Portkey-AI / gateway

A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.

https://portkey.ai/features/ai-gateway

MIT License

6k stars 410 forks source link

Feat: Provider Tests #494

Closed narengogi closed 1 month ago

narengogi commented 2 months ago

Title:

Tests for providers to make sure they are OpenAI compliant

Initial idea for this sprung from reading the very descriptive GitHub issue raised by @raulraja https://github.com/Portkey-AI/gateway/issues/116

Motivation: (optional)

Multiple provider's transformers are not OpenAI spec compliant, they fail to transform cases like user message content array
```
{
  role: 'user',
  content: [
    {
      type: 'text',
      text: 'Who is the greatest detective',
    },
  ],
}
```
Already caught 6 transformers failing from 12 I tested for the two test cases.

Related Issues: (optional)

116

TO-DO:

[X] Push config file
[For Later] Add flag to test only specific providers
[For Later] GitHub actions

narengogi commented 2 months ago

For your review @VisargD @sk-portkey

sk-portkey commented 1 month ago

@narengogi can you update to main. else lgtm