search

Portkey-AI / gateway

A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.
https://portkey.ai/features/ai-gateway
MIT License
5.87k stars 403 forks source link

Anthropic only uses Last System Message #457

Closed jpulec closed 5 days ago

jpulec commented 2 months ago

What Happened?

The way Portkey is set up, when sending multiple system messages to anthropic, only the last message is sent.

What Should Have Happened?

I'd propose that a better solution, since anthropic only allows a single system message to be provided, would be to concatenate all the provided system messages together, instead of only sending the last one.

If this seems reasonable, I can write the code for it.

Relevant Code Snippet

Note how system message is just wholesale overwritten in every iteration of param,messages:

https://github.com/Portkey-AI/gateway/blob/fe0899bbf1a0f65f8f751aca8fca1b0a4409acb4/src/providers/anthropic/chatComplete.ts#L174-L186

Your Twitter/LinkedIn

@jamespulec

vrushankportkey commented 2 months ago

Thank you @jpulec - this seems interesting and useful. Tagging @VisargD / @roh26it who can give the final go.

roh26it commented 1 month ago

Hi @jpulec - Sounds like the right approach, can you raise a PR for this. Would be happy to merge this.