search

jspsych / jsPsych

Create behavioral experiments in a browser using JavaScript
http://www.jspsych.org
MIT License
1.05k stars 679 forks source link

Audio-response plugin with speech recognition? (Web Speech API) #2404

Open becky-gilbert opened 2 years ago

becky-gilbert commented 2 years ago

Maybe we could use the Web Speech API to create a plugin that records spoken responses with automated speech recognition?

This could be based on the html-audio-response plugin and used to run tasks that require immediate spoken response scoring/categorization for dynamic elements. For instance, a Stroop task with spoken responses and correct/incorrect feedback, or a digit span task with spoken responses and a performance-based stop rule.

See speech color changer demo on this page: https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API

becky-gilbert commented 2 years ago

I should note that this API is experimental and not widely supported yet: https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API#browser_compatibility

jodeleeuw commented 2 years ago

This is fascinating. I was actually working on a very similar project with @tcharles-vassar last year, but we didn't know about this API. We were trying to use tensorflow.js to run a speech recognition model in the browser and perform a Stroop experiment with spoken responses. We were going to explore the possibility of training the model per user doing basic transfer learning and see what was possible for various experiments.