Speech to text

[allisonmorrell]

I'm experimenting with Chainlit and it is awesome. Kudos to you all.

Future feature request: I would like to incorporate speech to text in the user interface for my application.

Please label as enhancement.

[gruckion]

Would you be able to contribute this feature? I would be willing to coach you with adding this functionality.

[willydouhard]

I can't contribute in the short term but this is a good first issue (small and scoped to the UI). If any of you wants to make a PR (fork the repo and then open a PR against the Chainlit main branch) I would be more than happy to review it!

[allisonmorrell]

Would you be able to contribute this feature? I would be willing to coach you with adding this functionality.

Unfortunately, I don't know any TypeScript/React. That said, we are hoping to involve more people in our project including someone with more front-end experience so if someone were to outline the recommended approach at a very high level I might be able to find someone who can contribute.

[mmnasser2000]

Can someone assign this to me?

[sqrt676]

Hi @willydouhard , I am also trying to do, if i come up wuth something good, I will ask you to assign it to me and I will send the request to review it. Thanks !

[poojitharamachandra]

is this implemented?

[alimtunc]

poojitharamachandra

On it's way https://github.com/Chainlit/chainlit/pull/375

[poojitharamachandra]

do you know in which version this will be released?

[poojitharamachandra]

is this available in the latest version 0.7.604? if so how to use it? https://docs.chainlit.io/backend/config/features

[willydouhard]

Yes it is available in 0.7.604. To enable it just, update your config to:

[features]
prompt_playground = true
multi_modal = true
# Allows user to use speech to text
[features.speech_to_text]
    enabled = true
    language = "en-US"

[poojitharamachandra]

where should i make the above changes?

[allisonmorrell]

.chainlit/config.toml – discussion of features section here.

[poojitharamachandra]

thanks. where can I find this file in windows system?

[willydouhard]

It is located in the same directory you ran chainlit run ... at .chainlit/config.toml.

[poojitharamachandra]

thanks, now I see the option to record. But I am not able to record n post voice-based questions. Can u recommend a tutorial for that and for migration from 0.6.x to 0.7.x?

[willydouhard]

Here is the migration guide to 0.7.x

[poojitharamachandra]

is there an example on how to use speech?

[LeDuySon]

Hi @willydouhard, really nice feature. But i want to custom the model for speech to text part in python, how can i do that?

[willydouhard]

This is not supported yet. We rely on the browser APIs. Opened to contributions to optionally make it use a custom endpoint on the Chainlit Fast API server!

[poojitharamachandra]

is there an example on how to use speech? i am unable to record my voice from the UI

[willydouhard]

Might not work on all browsers since it relies on browser APIs. What browser do you use?

[poojitharamachandra]

I use edge. I see the button for voice recording on UI, but the question is not posted to the chat window as text. I even tried with mozilla, but I don't see option of voice recording at all. which browser do you recommend?

[willydouhard]

Can you try with chrome?

[poojitharamachandra]

due to security issues, I cannot use chrome on corporate network

[137particles]

Can you try with chrome?

I know I am late to the party but I just tested Speech To Text in 0.7.700 and 1.0.0RC3 and the latest Chrome (120) on MacOS. I can test in other environments (Windows, Ubuntu) and browsers if you have specific requests.

[willydouhard]

Does it work as intended @137particles ?

[137particles]

@willydouhard Yes. It correctly prompted for permission for the microphone and when granted I was able to click the icon and it captured what I said and then it did what appeared to be silence detection and turned the microphone off again.

[jitkoley-kasmo]

@137particles can you help to enable the mic in chainlit I have changed the config.toml file [features.speech_to_text] enabled = true
language = "en-US"

but it's not working.

[willydouhard]

What browser do you use?

[Girrajjangid]

Same thing happening with me.

[features.speech_to_text]
enabled = true
language = "en-US"

I edited config.toml file. I am using Chrome browser. ✅ I gave microphone permission as well. ✅ I can see microphone button as well. When I click on microphone button it start and automatically stop within 2 second. I don't know whether it record something. Because there is no log captured.

@willydouhard Can please give me some direction? And thank you man this library is awesome. 🔥🔥Very easy to use.

[remzicam]

It's happening because of this line in the front-end code. It stops after 3 (I guess it should be 2) seconds of silence. @willydouhard Could the timer be extended by at least 10 seconds, or alternatively, could an option be provided to allow users to set their own timer duration?

[melmansouri1]

Any update guys ?

[melmansouri1]

Any news ?

[didlawowo]

It's happening because of this line in the front-end code. It stops after 3 (I guess it should be 2) seconds of silence. @willydouhard Could the timer be extended by at least 10 seconds, or alternatively, could an option be provided to allow users to set their own timer duration?

yeap, but it doesn't record anything on brave, seems working on chrome

[shaungt1]

Same thing happening with me.

[features.speech_to_text]
enabled = true
language = "en-US"

I edited config.toml file. I am using Chrome, Firefox, Edge, and Brave browser. ✅ I gave microphone permission as well. ✅ I can see microphone button as well.

On local development I can use it works just fine across all browsers but once I deploy chainlit from ubunut when I press the mic button it just turns off and doesnt accept any speach.

in the dev-tools network tab I get a confg and language-US object back on ubuntu

{
    "requireLogin": null,
    "passwordAuth": false,
    "headerAuth": false,
    "oauthProviders": []
}

{
    "components": {
        "atoms": {
            "buttons": {
                "userButton": {
                    "menu": {
                        "settings": "Settings",
                        "settingsKey": "S",
                        "APIKeys": "API Keys",
                        "logout": "Logout"
                    }
                }
            }
        },
        "molecules": {
            "newChatButton": {
                "newChat": "New Chat"
            },
            "tasklist": {
                "TaskList": {
                    "title": "🗒️ Task List",
                    "loading": "Loading...",
                    "error": "An error occured"
                }
            },
            "attachments": {
                "cancelUpload": "Cancel upload",
                "removeAttachment": "Remove attachment"
            },
            "newChatDialog": {
                "createNewChat": "Create new chat?",
                "clearChat": "This will clear the current messages and start a new chat.",
                "cancel": "Cancel",
                "confirm": "Confirm"
            },
            "settingsModal": {
                "settings": "Settings",
                "expandMessages": "Expand Messages",
                "hideChainOfThought": "Hide Chain of Thought",
                "darkMode": "Dark Mode"
            },
            "detailsButton": {
                "using": "Using",
                "running": "Running",
                "took_one": "Took {{count}} step",
                "took_other": "Took {{count}} steps"
            },
            "auth": {
                "authLogin": {
                    "title": "Login to access the app.",
                    "form": {
                        "email": "Email address",
                        "password": "Password",
                        "noAccount": "Don't have an account?",
                        "alreadyHaveAccount": "Already have an account?",
                        "signup": "Sign Up",
                        "signin": "Sign In",
                        "or": "OR",
                        "continue": "Continue",
                        "forgotPassword": "Forgot password?",
                        "passwordMustContain": "Your password must contain:",
                        "emailRequired": "email is a required field",
                        "passwordRequired": "password is a required field"
                    },
                    "error": {
                        "default": "Unable to sign in.",
                        "signin": "Try signing in with a different account.",
                        "oauthsignin": "Try signing in with a different account.",
                        "redirect_uri_mismatch": "The redirect URI is not matching the oauth app configuration.",
                        "oauthcallbackerror": "Try signing in with a different account.",
                        "oauthcreateaccount": "Try signing in with a different account.",
                        "emailcreateaccount": "Try signing in with a different account.",
                        "callback": "Try signing in with a different account.",
                        "oauthaccountnotlinked": "To confirm your identity, sign in with the same account you used originally.",
                        "emailsignin": "The e-mail could not be sent.",
                        "emailverify": "Please verify your email, a new email has been sent.",
                        "credentialssignin": "Sign in failed. Check the details you provided are correct.",
                        "sessionrequired": "Please sign in to access this page."
                    }
                },
                "authVerifyEmail": {
                    "almostThere": "You're almost there! We've sent an email to ",
                    "verifyEmailLink": "Please click on the link in that email to complete your signup.",
                    "didNotReceive": "Can't find the email?",
                    "resendEmail": "Resend email",
                    "goBack": "Go Back",
                    "emailSent": "Email sent successfully.",
                    "verifyEmail": "Verify your email address"
                },
                "providerButton": {
                    "continue": "Continue with {{provider}}",
                    "signup": "Sign up with {{provider}}"
                },
                "authResetPassword": {
                    "newPasswordRequired": "New password is a required field",
                    "passwordsMustMatch": "Passwords must match",
                    "confirmPasswordRequired": "Confirm password is a required field",
                    "newPassword": "New password",
                    "confirmPassword": "Confirm password",
                    "resetPassword": "Reset Password"
                },
                "authForgotPassword": {
                    "email": "Email address",
                    "emailRequired": "email is a required field",
                    "emailSent": "Please check the email address {{email}} for instructions to reset your password.",
                    "enterEmail": "Enter your email address and we will send you instructions to reset your password.",
                    "resendEmail": "Resend email",
                    "continue": "Continue",
                    "goBack": "Go Back"
                }
            }
        },
        "organisms": {
            "chat": {
                "history": {
                    "index": {
                        "showHistory": "Show history",
                        "lastInputs": "Last Inputs",
                        "noInputs": "Such empty...",
                        "loading": "Loading..."
                    }
                },
                "inputBox": {
                    "input": {
                        "placeholder": "Type your message here..."
                    },
                    "speechButton": {
                        "start": "Start recording",
                        "stop": "Stop recording"
                    },
                    "SubmitButton": {
                        "sendMessage": "Send message",
                        "stopTask": "Stop Task"
                    },
                    "UploadButton": {
                        "attachFiles": "Attach files"
                    },
                    "waterMark": {
                        "text": "Built with"
                    }
                },
                "Messages": {
                    "index": {
                        "running": "Running",
                        "executedSuccessfully": "executed successfully",
                        "failed": "failed",
                        "feedbackUpdated": "Feedback updated",
                        "updating": "Updating"
                    }
                },
                "dropScreen": {
                    "dropYourFilesHere": "Drop your files here"
                },
                "index": {
                    "failedToUpload": "Failed to upload",
                    "cancelledUploadOf": "Cancelled upload of",
                    "couldNotReachServer": "Could not reach the server",
                    "continuingChat": "Continuing previous chat"
                },
                "settings": {
                    "settingsPanel": "Settings panel",
                    "reset": "Reset",
                    "cancel": "Cancel",
                    "confirm": "Confirm"
                }
            },
            "threadHistory": {
                "sidebar": {
                    "filters": {
                        "FeedbackSelect": {
                            "feedbackAll": "Feedback: All",
                            "feedbackPositive": "Feedback: Positive",
                            "feedbackNegative": "Feedback: Negative"
                        },
                        "SearchBar": {
                            "search": "Search"
                        }
                    },
                    "DeleteThreadButton": {
                        "confirmMessage": "This will delete the thread as well as it's messages and elements.",
                        "cancel": "Cancel",
                        "confirm": "Confirm",
                        "deletingChat": "Deleting chat",
                        "chatDeleted": "Chat deleted"
                    },
                    "index": {
                        "pastChats": "Past Chats"
                    },
                    "ThreadList": {
                        "empty": "Empty...",
                        "today": "Today",
                        "yesterday": "Yesterday",
                        "previous7days": "Previous 7 days",
                        "previous30days": "Previous 30 days"
                    },
                    "TriggerButton": {
                        "closeSidebar": "Close sidebar",
                        "openSidebar": "Open sidebar"
                    }
                },
                "Thread": {
                    "backToChat": "Go back to chat",
                    "chatCreatedOn": "This chat was created on"
                }
            },
            "header": {
                "chat": "Chat",
                "readme": "Readme"
            }
        }
    },
    "hooks": {
        "useLLMProviders": {
            "failedToFetchProviders": "Failed to fetch providers:"
        }
    },
    "pages": {
        "Design": {},
        "Env": {
            "savedSuccessfully": "Saved successfully",
            "requiredApiKeys": "Required API Keys",
            "requiredApiKeysInfo": "To use this app, the following API keys are required. The keys are stored on your device's local storage."
        },
        "Page": {
            "notPartOfProject": "You are not part of this project."
        },
        "ResumeButton": {
            "resumeChat": "Resume Chat"
        }
    }
}

these settings are exaclty same for both loca and deployed version Is there any guidiance on how make this STT work on Ubuntu for a deployed instiance?