Key commands for vision impaired

freetimekate commented 1 year ago

Thank you for creating this amazing tool. It's a game changer for those of us using voice to interact with their PC. Would you please consider adding a key commands that would trigger the start and skip functionality? This would enable us a screen free experience. Thanks again for all your work!

C-Nedelcu commented 1 year ago

hi freetimekate,

do you mean like keyboard shortcuts like CTRL+ALT+something? this is pretty easy to add yeah, what shortcuts would be helpful?

I could see the following:

muting the mic
muting the bot
skipping the current message

Any further suggestions? This could make for a good update for V1.7

freetimekate commented 1 year ago

Hi C-Nedelcu,

That's it exactly! Just like using CONTL+R to refresh the page. So cool to hear it could be relatively easy. Thinking about this in detail, for a low-vision or screen-free experience, I would recommend a keyboard shortcut for the following:

Actuate start (depressing ENTER or equivalent). This way you can refresh the page with CTRL+R, then start the tool without the mouse
Stop/skip bot (as discussed)
Toggle Mute on/off (setting the default mute state of the mute button as discussed)
Mute override keyboard shortcut (press and hold to temporarily unmute/mute). This is particularly helpful for users who can't see the icons and are in environments where others may be speaking. Simply set the default to mute(or unmute) and press and hold a keyboard shortcut any time you'd like to input with voice(or speak without inputting if mute is toggled on). It's easy enough to toggle the buttons when you can see the icons but this feature would be very helpful for low vison and screen free applications.
In an ideal world there would be a voice input mode. With one key command, the page is refreshed, start is actuated, and the microphone is toggled to mute. There could be mention keyboard shortcut in your settings page.

The low-vision user would then use two keyboard shortcuts primarily; skip/stop key command to interrupt, and press and hold mute override key(s) to input with voice.

Now that I've said that I'm realizing, if you're intending to speak, then you probably don't want to be talked over, so the mute override key should also skip the current message. That actually be slick. More like natural conversation.

The user would then only have a single keyboard shortcut to operate the interface if low-vision. I could also see this used in headless applications for communicating with chat GPT without a screen at all, just a single voice input button. Just press and told to input with voice.

Would love to hear your thoughts. Thanks again for creating such a helpful tool and considering these updates for low-vision and screen-free users! Looking forward to your next update whatever you choose to do!

graham-archer commented 1 year ago

Hi there, I just wanted to say that this wouldn't only be useful for the visually impaired but I think it would be great to have a shortcuts in general, for all users. If the shortcuts were user customisable that would be even better! It's an excellent product thanks so much for making.

drorm commented 1 year ago

Hi @freetimekate, I created a similar app, Leah: https://github.com/drorm/leah https://chrome.google.com/webstore/detail/leah-language-educator-an/bolaopgaickihobdlddefmdffopigkmh that is focused on language learning, but I'd love to explore adapting or creating a different version that's focused on folks that are vision impaired. Other than shortcuts, is there any other functionality that you'd be interested in?

C-Nedelcu commented 1 year ago

Hi @freetimekate, I created a similar app, Leah: https://github.com/drorm/leah https://chrome.google.com/webstore/detail/leah-language-educator-an/bolaopgaickihobdlddefmdffopigkmh that is focused on language learning, but I'd love to explore adapting or creating a different version that's focused on folks that are vision impaired. Other than shortcuts, is there any other functionality that you'd be interested in?

This is highly interesting and exciting (much more so than my own extension which I consider to be basically a gimmick). I'm amazed to discover how people find new uses for ChatGPT. Its capabilities in terms of translation, grammar, spell check, and everything that has to do with languages in general, are often overlooked in the press. Truthfully when it comes to translations ChatGPT performs incredibly well !

I'm sorry I haven't been making progress on keyboard shortcuts recently. I haven't had time to dedicate to improving Talk-to-ChatGPT lately. I'd love it if there were some pull requests from other developers, anyone's welcome to contribute.

drorm commented 1 year ago

@C-Nedelcu, I'm so glad to see your reaction since I was a little reluctant to post about a "competing" product, which is funny since both are totally open source :-). ChatGPT does really well, better than Google translate, with mainstream languages such as as English, French and Spanish, as far a I can tell. With more obscure languages, such as Hebrew, it still has rough edges. I'm not surprised since there are a lot less docs for ChatGPT to train, and I expect it to improve over time.

I wish I had figured out to use your product and build on it earlier, but at this point, I'm used to my stack: Angular, typescript, etc. So will continue there.

Happy though to answer some of the obvious questions in the issues.

C-Nedelcu commented 1 year ago

I have added keyboard shortcuts in v2.0

ALT+SHIFT+S: Start Talk-To-ChatGPT ALT+SHIFT+H: suspend/resume speech recognition (Hush) ALT+SHIFT+V: suspend/resume bot's voice (Voice) ALT+SHIFT+L: skip current message (Leap)

The choice of key combination was particularly difficult when you realize that Chrome already uses a LOT of these. All the ones I wanted were already taken, basically.

C-Nedelcu / talk-to-chatgpt

Key commands for vision impaired #25