Open artmoskvin opened 4 months ago
Here's the plan for implementing response streaming in the AI
class:
Refactor the AI
class: Modify the AI
class to include a method for streaming responses. This method should take the same parameters as the call
method but return a generator instead of a string.
Update the call
method: The call
method should be updated to use the new streaming method. This will ensure that all existing code that uses the call
method will still work as expected.
Update the BaseChatModel
interface: The BaseChatModel
interface should be updated to include a method for streaming responses. This method should take the same parameters as the existing method but return a generator instead of a string.
Update all BaseChatModel
implementations: All classes that implement the BaseChatModel
interface should be updated to include the new streaming method. This will likely involve refactoring the existing method to use the new streaming method.
Test the new implementation: Finally, the new streaming implementation should be thoroughly tested to ensure it works as expected. This should include both unit tests and integration tests.
Currently,
AI
class handles only batch completions. To improve UX it should also support streaming.