search

barry802 / FD-APAC-AI-Project

FD APAC AI Project
2 stars 1 forks source link

Improve latency #6

Open roudra-das opened 4 months ago

roudra-das commented 4 months ago

Need to improve response latency.

Possible solutions:

roudra-das commented 4 months ago

I'l try implement streaming i.e. response is output in sync with ChatGPT rather than waiting for the entire output

@kelvouttt do you think caching you could implement caching?