search

leetcode-mafia / cheetah

Mac app for crushing remote tech interviews with AI
Creative Commons Zero v1.0 Universal
4.08k stars 297 forks source link

Answers take long to generate #12

Closed tmcinally closed 1 year ago

tmcinally commented 1 year ago

Running: release 1.00, not built locally

Hi,

I have everything set up and transcribing speech to text seems to be working. I have been able to generate answers on some occasions, but i am having issues with this.

My main issue is that when i press "answer", the answers take very long to generate, and sometimes the answers are not generated at all. It seems that this might be a general issue, since it seems to take a while in the demo videos as well. I would say it is so slow that the app isn't really useable at this point.

Is there any setting i can change in order to speed up the time it take to generate an answer? Maybe i need to build the app locally?

Good job on the app so far by the way.

leetcode-mafia commented 1 year ago

The root cause is simply that the GPT-4 API can be very slow during periods of high demand. This is a general problem with GPT-4 at the moment.

Two ideas to consider:

  1. You can force the app to use GPT-3.5, which will be faster but less accurate:
defaults write org.phrack.Cheetah useGPT4 -bool no
  1. We could use the streaming API to show each token as it's generated, like ChatGPT does. This would make things feel more responsive. (I would gladly accept a PR that implements this!)
tmcinally commented 1 year ago

Hmm okay, thanks. I think maybe there was some other bug going on with me. Most of the time when i pressed "answer" there was just no result at all. I downloaded the newest version and it works much better now