The current brute-force implementation of streaming, where each chunk of output from gpt is saved to the database works but slams the database really hard and won't scale. Change the conversation job to stream to the client using cable ready and then save the entire output once it is generated fully.
The current brute-force implementation of streaming, where each chunk of output from gpt is saved to the database works but slams the database really hard and won't scale. Change the conversation job to stream to the client using cable ready and then save the entire output once it is generated fully.