Stream implementation

[adrienbanse]

This PR is a draft for https://github.com/dataforgoodfr/12_genai_impact/issues/16

• [x] OpenAI streams
• [x] OpenAI streams & async
• [x] OpenAI tests
• [x] Anthropic streams
• [x] Anthropic streams & async
• [x] Anthropic tests
• [x] Mistral streams
• [x] Mistral streams & async
• [x] Mistral tests

[adrienbanse]

Concerning Anthropic, for now streams work with

from genai_impact import Tracer
from anthropic import Anthropic

Tracer.init()
client = Anthropic()

stream = client.messages.stream(
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
    model="claude-2.0",
)

print(stream.impacts)

But not with

from genai_impact import Tracer
from anthropic import Anthropic

Tracer.init()
client = Anthropic()

with client.messages.stream(
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
    model="claude-2.0",
) as stream:
    for text in stream.text_stream:
        print(text)
    print(stream.impacts)

But I have an idea on how to solve this

@samuelrince If I'm not wrong, although text is sent, Anthropic sends the output_tokens only at the end of the stream, so we can only compute it for the whole stream, unlike chunks with OpenAI

[LucBERTON]

I implemented stream and async stream for mistralai, please check and let me know if you think it's OK

[LucBERTON]

Regarding Anthropic,

I tested Andrien's code but I think it's not returning any impacts value in "stream".

I have to say that I'm not sure I understand how to use a wrapper with a context manager (with ... as stream) as we're not just calling a function like what we do with openai or mistralai :

    with client.messages.stream(
        max_tokens=1024,
        messages=[{"role": "user", "content": "Hello"}],
        model="claude-3-haiku-20240307",
    ) as stream:

[adrienbanse]

Anthropic streams now work with both options in https://github.com/dataforgoodfr/12_genai_impact/pull/28#issuecomment-2038166410

Remains to make it work with

stream = client.messages.create(
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Hello, Claude",
        }
    ],
    model="claude-3-opus-20240229",
    stream=True,
)

for event in stream:
    print(event.type)

Here the type of stream is Anthropic.Stream

[samuelrince]

Hello on my side this does not seem to work for Anthropic. If the Tracer is initialized, the text is not printed.

from genai_impact import Tracer
from anthropic import Anthropic

Tracer.init()
client = Anthropic()

with client.messages.stream(
        max_tokens=100,
        messages=[{"role": "user", "content": "Hello World!"}],
        model="claude-3-haiku-20240307",
) as stream:
    for text in stream.text_stream:
        print(text)
    print(stream.impacts)

[adrienbanse]

Found a way to make it work with

from genai_impact import Tracer
from anthropic import Anthropic

Tracer.init()
client = Anthropic()

with client.messages.stream(
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
    model="claude-2.0",
) as stream:
    for text in stream.text_stream:
        print(text)
    print(stream.impacts)

but not with the previous option (stream = ...). I believe that this makes more sense.

Can you test on your side @samuelrince?