Support for Claude 3 models

[teremterem]

Currently it looks like logs (captured requests and responses) for the new Claude 3 models don't show up at https://promptlayer.com/workspace/.... Logs for Claude 2 show up fine.

To test Claude 2 I used promptlayer.anthropic.Anthropic.completions.create and it worked (logs showed up).

To test Claude 3 I use promptlayer.anthropic.AsyncAnthropic.messages.create/promptlayer.anthropic.AsyncAnthropic.messages.stream and the logs don't show up in PromptLayer.

I did follow all the instructions to configure the python promptlayer client properly.

[Jped]

@teremterem I can confirm that I am seeing this bug with AsyncAnthropic, looking into a solution

[Jped]

Hey @teremterem can you pull the most recent version of the python library and try again?

[teremterem]

Hey @Jped thank you for such a quick reaction! Unfortunately the logs don't show up with 0.5.3 either. It gives me this warning when I try:

WARNING: While logging your request PromptLayer had the following error: Object of type AsyncMessageStreamManager is not JSON serializable

[teremterem]

The logs now do show up when I turn off token streaming, though. So the problem remains only when token streaming is on.

[Jped]

@teremterem my bad, forgot to check for streaming. Looking into that right now.

[Jped]

Hey @teremterem this is a bit of a bigger issue than i expected. I dont think we have been supporting streaming for anthropic ever since they have moved to messages. Is this a blocking bug for you? Just want to see how to prioritize this

[teremterem]

Hi @Jped I will keep using Anthropic in streaming mode (because it contributes to better UX) and for awhile I'll be ok without the logs, but there will be a point in time when having logs is important to me again. I don't have a specific timeline just yet and I also don't want to put pressure on you, so please take your time. Thanks for reacting to my issue so soon in the first place!

[Jped]

hey @teremterem just pushed an update and it works in my testing.

It does not work for the text_stream example anthropic gives in their docs, but it does work for this code snippet:

async def main() -> None:
    stream = await client.messages.create(
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Hello, Claude",
        }
    ],
    model="claude-3-opus-20240229",
    stream=True,
    )
    async for event in stream:
        # print(event)
        pass

LMK if this works.

[teremterem]

Hey @Jped, I finally had a chance to check it out. Yes, it works, thank you!

Would you by any chance be able to support the text_stream version too some time in the future? The text_stream based code looks cleaner because there is no need to think about different event types and which ones to filter out (plus, since text_stream is in the docs, it will most likely be the first thing everyone will try anyways - only to discover that it is not logged in your platform).

I haven't checked how easy it is to get text generation metadata (token count etc.) with the non-text_stream version yet, but if it happens to be messy, I'll probably have to go back to text_stream.