search

barry802 / FD-APAC-AI-Project

FD APAC AI Project
2 stars 1 forks source link

Improve latency #6

Open roudra-das opened 1 month ago

roudra-das commented 1 month ago

Need to improve response latency.

Possible solutions:

roudra-das commented 1 month ago

I'l try implement streaming i.e. response is output in sync with ChatGPT rather than waiting for the entire output

@kelvouttt do you think caching you could implement caching?