krschacht
commented
2 months ago - [x] SMALL BUT EASY: The stop icon flashes briefly after submitting the first message (this is because it's doing a Turbo Drive navigate rather than a Turbo Morph, see if I can detect this). NO EASY WAY TO FIX
- [x] SMALL BUT EASY: When the user is enabling the mic and is asked about enabling the screen sharing, the mic actually turns on before they've answered the screen sharing question. Figure out where I missed an await.
- [x] IMPORTANT: Add sound effects so the user can tell when the AI is waiting, thinking, etc.
- [x] Decide how I should handle when markdown code blocks appear in the response
- [x] FUTURE: When there is a lot of background noise (e.g. music playing) then the system never detects any silence. Maybe we can establish a baseline noise level and re-define "silence" as a return to baseline noise level and/or we can submit responses after X seconds even if we don't detect any silence.
- [x] FUTURE: Right now a wake word (e.g. the assistant's name) is not needed. By default, the assistant is always listening unless you've dismissed it by by saying "Hold on Samantha". However, more lifelike would be for the assistant to dismiss itself after X minutes of you not talking to it. If someone is sitting in the room with us but we started ignoring them for ~30 minutes then we naturally re-address them if we start talking again.
- [ ] SMALL: It keeps playing the typing sound when you've suspended it. Also, the typing sound needs to be randomized a bit more to sound realistic. It's too repetitive.
- [ ] IMPORTANT: Wrap the system prompt with voice-specific instructions
- [ ] IMPORTANT: Route all the requests through a proxy
- [ ] IMPORTANT: Decrease latency so the user gets a faster reply. My best idea is this. Current it waits for 1.8s of silence before submitting the response. Instead, only wait for maybe 0.2 - 0.5s of silence, after that assume go ahead and submit the message to the API, but then if we detect the user starts talking again before we've received and started playing back the response edit the message we submitted with these additional words and re-submit it (this will take advantage of conversation branching which is already in place). Note: we need to confirm the previous reply job will be cancelled when this revised message is submitted.
- [ ] IMPORTANT: Add full test coverage and get CI to run the javascript tests
- [ ] FUTURE: Is there a heuristic for knowing when it’s overhearing a conversation between two people vs when you’re talking to it?
lumpidu