mit-han-lab / TinyChatEngine

TinyChatEngine: On-Device LLM Inference Library
https://mit-han-lab.github.io/TinyChatEngine/
MIT License
634 stars 59 forks source link

Made dictation concurrent with token generation #61

Closed Jiminator closed 9 months ago

Jiminator commented 9 months ago

Description

Dictation of the voice chat will no longer wait for the model to finish generating a response but instead will be concurrent to token generation. This is done by using multithreading and separating the output into phrases separated by delimiters such as punctuation.

Changes Made

Video

https://www.youtube.com/watch?v=mv2y9HRS1mE&feature=youtu.be

Checklist

Please review and check off these items before submitting your pull request:

Reviewers

@RaymondWang0