mallorbc
commented
10 months ago Thanks! But there is an issue
Thanks for the PR! Sorry It took so long to review. I really like this however, with my testing, there are issues with the PR with regards to the listen_loop functionality.
If no phrase_timeout is given, it will not transcribe in "real time" anymore. That is ok and is to be expected. Giving a value to that argument mostly fixes the issue.
However, during the listen_loop method, during transcription, any words said during that time will be lost. One can easily see this by giving a phrase timeout value of say 2 and seeing that at the end of the two seconds, transcription will start and a word or two will be missed during this time. The current implementation on the main branch does not miss any words.
What I think the problem is
The reason for this I think can be attributed to the fact that the microphone is no longer listening in the background so during the code here: https://github.com/sankhadeepdutta/whisper_mic/blob/a0be2a17d11da075c561a9261de8df7ea839a27c/whisper_mic/whisper_mic.py#L88-L91
The microphone is no longer taking in audio data and has to wait for the transcription to finish before it starts listening again.
Other than this issue, I think that your implementation is great. For the simple listen method, it is superior. And of course, the comments, and cleaning up the code is much appreciated.
Ideas and next steps
I am going to make this a branch and work on it when I get a chance. At the same time, feel free to make additional changes to address these issues if you are so inclined.
I think using your method for the listen method is better as I already said, but either threading or listening in the background is needed for the real-time transcription.
Perhaps we use a variation of your solution for the listen method and a variation of the existing solution for the listen_loop method.
sankhadeepdutta
SunnyOd
Fixes #46
Changes made:
All the methods related to the implementation details of the WhisperMic class have been made private. Only the methods necessary for the user are public.
Used the listen() method of Recognizer class from the speech_recognition package, instead of listen_in_background() method. It will make sure the listen() method of the WhisperMic class works as intended. It will wait for the user to complete his/her voice input and then take further action. If the user provides the values for the optional parameters timeout and phrase_time_limit, the listen() method will work accordingly. timeout: determines how long the listen() method will wait for audio input. phrase_time_limit: specifies how long the recognizer should wait for a spoken phrase to be completed. The default is “None” for both.
[Important] A new method, __record_handler() has been added which implements the record() method of the Recognizer class in speech_recognition package. The key differences between the record() and listen() methods of the Recognizer class in the speech_recognition package are: record() captures audio for a fixed duration specified by the user. listen() captures audio in real-time and automatically detects the end of speech using silence detection Looking at the functionality, I believe the current implementation of the listen() method of the WhisperMic class wants to incorporate the record() method actually. So I have added a new record() method, which calls the __record_handler() method and transcribes the audio input. It has the optional parameters: duration and offset. Duration: specifies how long the recording should continue. Offset: specifies how long to wait before actually starting the recording. The default is “None” for both. Reference: https://github.com/Uberi/speech_recognition/blob/master/reference/library-reference.rst
The record_callback() method has been renamed to __record_load() method whose task is to load the raw version of the recorded audio to the audio_queue queue.
A new method, __listen_handler() has been added whose task is to handle the audio record process, load the audio data to the audio_queue queue, and finally transcribe the audio data. The output is stored in the result_queue queue. This method has been used both for implementing the listen() and the listen_loop() method.
6: Added exception handling for __listen_handler() method, as when the user sets a value for the “timeout” parameter in listen() method, it will raise an exception if the user doesn’t start speaking within the given time.
The task of recording the audio from the mic has been shifted from the setup_mic() method to the __listen_handler() method. The setup_mic() method should be associated only to set up the mic properties. The program would start recording only after the user calls either the listen() or listen_loop() method.
timeout, phrase_time_limit parameters has been added to the listen_loop() method as well.