search

naeruru / mimiuchi

a free, customizable, osc capable speech-to-text interface for relaying text to different types of applications
https://mimiuchi.com
GNU General Public License v3.0
40 stars 4 forks source link

Add support for real-time text in "Type a message" #40

Closed mdrejhon closed 1 day ago

mdrejhon commented 1 month ago

Hello!

I'm a deaf software developer, who uses both transcription (for incoming text) and keyboard (for outgoing text).

I've been immensely enjoying the ability to use mimiuchi as a free alternative to Otter or other paid services which I also use. I like the fact that mimiuchi supports WebHook and WebSockets.

Thanks to "Type a message", mimiuchi conveniently can be programmed as a browser-based keyboard for another device (with a bit of programming glue). I also like using mimiuchi to turn a phone/iPad into a portable keyboard for a big screen.

Use case: Mimiuchi acts as a substitute for a bluetooth keyboard

Allows guests to bring their own device, to use as a keyboard for a main communications screen. When no spare keyboard is available.

Or that we're doing chatting on-the-go across the table (like a Ubiduo assistive device) privately between two offline devices (mimiuchi works offline once cached). Or multiple chatters (one at a time) chatting in real time to a central big screen (e.g. living room TV or conference room projector) in front of them.

Basically any iPhone/Android/iPad/laptop/etc can become a portable guest keyboard for another device such as a main big chat screen (on TV). Transcriptions still work but some people prefer to type. The guest typist sitting next to me, can use their device to send messages to the big screen (through mimiuchi) which works instead of a Bluetooth keyboard.

However, it's not as realtime-feeling as a bluetooth keyboard typing to a laptop connected directly to a big screen. It's nice giving people a choice to speak or type...

So as a workaround for this use case:

Is it possible to add a realtime mode to the "Type a Message" where keypresses are sent one by one in real time? Or even as batches every 0.25 second or 0.5 seconds? (A configurable time-based buffer to rate-limit webhook/websocket requests).

Basically an optional "Real time typing" mode where keypresses are sent individually (or almost).

mdrejhon commented 1 month ago

Some UX thoughts:

One unresolved question is what to do with message edits if doing it via "Type a Message" box. Maybe use the same autocorrect protocol (text-replace). Or, if modified to type directly into the main text area, then only backspace would be supported.

Whichever is easier to adapt mimiuchi architecture to double as a realtime "web keyboard" client for a remote chatting screen or central chatting screen.

naeruru commented 1 month ago

Sure, I can work on this! I had some ideas for a separate, typing focused home screen, and I’ll mess with that more this week and next. I like your simple idea of just making the text window a text area of some sort

mdrejhon commented 1 day ago

Fantastic. Thanks, @naeruru!

When will this version be put up at www.mimiuchi.com? I'll give it a test.

This should produce some really interesting "use your smartphone as a web-based keyboard" and "guest keyboard" use cases.

And the bonus, you can use either realtime dictation or realtime typing for all the above.

One person can type (in mimiuchi), another person can speak (in mimiuchi), straight into a 3rd app (just 100 lines of code) that is a splitscreen like old Unix Talk, BBS Talk or a Ubiduo

With this simple change, I suspect Mimiuchi just became probably the world's first general-purpose platform-independent "turn any device into a keyboard" client. Use line-keyboard, realtime-keyboard, or voice dictation, your choice. Very accessible!

And you can have multiple devices running multiple copies of mimiuchi to the same screen for collaborative use cases.

Your modification opens quite a lot of interesting assistive and non-assistive use cases.