Open swiftugandan opened 5 months ago
Thanks for filing @swiftugandan. Would you be open to contributing a PR? Happy to walk you through where to make changes in the code.
Yes sure, where should I look?
We likely need to add logic in fetchOpenAI
similar to how we call translateParams
on Anthropic.
Currently translateParams
operates in terms of ModelFormat
, so I think we need to create something like translateProviderParams
that does something similar in terms of ModelEndpointType
. To keep things simple, I would create a map that specifies which params are disallowed by specific endpoint types (in this case, groq
), and then check this logic in fetchOpenAI()
.
BTW, are you passing these params in and hoping that they're stripped, or are they being inserted automatically somewhere? If the latter, maybe to start, we remove these parameters if they're specified, but null
, and throw an error if they are non-null?
I think they are being added as default params automatically, let me run the proxy to verify what is being sent
I have successfully run the node proxy, but when I send a groq message, I am getting an error Failed to lookup api key: Failed to validate JWT: Error: JWT string does not consist of exactly 3 parts (header, payload, signature)
from https://github.com/braintrustdata/braintrust-proxy/blob/b87c35a63db341e88b8ed1216c0e7e457868c5b5/apis/node/src/login.ts#L19
I reproduced the error using curl.
➜ braintrust-proxy git:(main) ✗ curl -X POST \ -H "Authorization: Bearer gsk_L..." \ -H "Content-Type: application/json" \ -d '{ "model": "mixtral-8x7b-32768", "org_name": "undefined", "mode": "full" }' \ "https://www.braintrustdata.com/api/secret"
How can I get past this step?
The node proxy does not currently support "api token passthrough", which the cloudlfare one does. Could you debug locally using the cloudflare proxy? Just cd into apis/cloudflare
and run:
pnpm install
cp wranger-template.toml wrangler.toml
npx wrangler start
and it should work
it turns out no additional fields are being injected. The problem is with the messages array that is being sent.
groq expects messages: [{"role": "user", "content": "some string"}]
but I was sending the OpenAI format messages: [{"role": "user", "content": [{ "type": "text", "text":"some string" }]}]
... do we need to add support for this format?
Very interesting. We could do something similar to what I suggested above, which per-ModelEndpointType
we boil down the chat messages into a simpler format.
Groq does not support some of the fields that are set in the body see: https://console.groq.com/docs/openai
The openai js library automatically adds some of these fields, resulting in 400 errors.
Please help trim these out for groq requests