Dictation of the voice chat will no longer wait for the model to finish generating a response but instead will be concurrent to token generation. This is done by using multithreading and separating the output into phrases separated by delimiters such as punctuation.
Changes Made
updated voicechat.cc to accommodate changes made in previous commits.
changed the generate function to only store phrases of the output instead of the whole response, and to dictate the phrase as soon as it is done generating.
Description
Dictation of the voice chat will no longer wait for the model to finish generating a response but instead will be concurrent to token generation. This is done by using multithreading and separating the output into phrases separated by delimiters such as punctuation.
Changes Made
Video
https://www.youtube.com/watch?v=mv2y9HRS1mE&feature=youtu.be
Checklist
Please review and check off these items before submitting your pull request:
Reviewers
@RaymondWang0