Microphone input

[fonsp]

I saw this awesome tweet and I want it too!

https://twitter.com/Mr_Auk/status/1289080703376924675

Streaming an entire audio stream might not work well, but constantly sending short audio snippets (say, a 10ms recording every 100ms) would definitely work! But maybe all of it can be sent live?

[mthelm85]

From Issue #14:

Thanks!

Are you interested in writing another one? How about #16 😊

It would be very cool if you could make something in Vue to include here - do you have any ideas?

I'm certainly willing to give it a go! I'm not sure though exactly what's going on there in the notebook in the Tweet...? What exactly is the goal or desired outcome? Is it just the ability to record audio from the mic?

[mthelm85]

This looks like a pretty good roadmap:

https://developers.google.com/web/fundamentals/media/recording-audio

I'll play around with it this weekend and see if I can come up with something : )

It would probably be good to figure out what kind of inputs are expected in the JuliaAudio family of packages:

https://github.com/JuliaAudio

I know nothing at all about this but I assume it would be good to be able to take audio from the user's mic and encode it in a format that can be consumed by one of those packages so that people can do whatever they do with that kind of thing ; )

[fonsp]

Cool! Let me know

[mthelm85]

I made a tiny little bit of progress:

Here's the code so you can just copy/paste and try it out:

@bind audio HTML("""
<audio id="player"></audio>
<script>
const player = document.getElementById('player')

const handleSuccess = function(stream) {
  const context = new AudioContext()
  const analyser = context.createAnalyser()
  const source = context.createMediaStreamSource(stream)

  source.connect(analyser)
  analyser.connect(context.destination)
  analyser.fftSize = 2048;
  var bufferLength = analyser.frequencyBinCount
  var dataArray = new Uint8Array(bufferLength)
  analyser.getByteTimeDomainData(dataArray)
  player.value = dataArray
  player.dispatchEvent(new CustomEvent("input"))
}

navigator.mediaDevices.getUserMedia({ audio: true, video: false })
  .then(handleSuccess)
</script>
""")

I am completely out of my league with this...I have no idea what the output means but it seems pretty useless right now...

[fonsp]

Oh this is going great! The Dict output should be fixed if you update Pluto to the latest version - it now sends Uint8Array() back as a Array{UInt8,1} instead of this weird behaviour.

[fonsp]

Oh but you are doing a fourier transform on the microphone data right? That's funky

[mthelm85]

This seems to be doing something more interesting:

@bind audio HTML("""
<audio id="player"></audio>
<script>
const player = document.getElementById('player')

const handleSuccess = function(stream) {
  const context = new AudioContext()
  const analyser = context.createAnalyser()
  const source = context.createMediaStreamSource(stream)

  source.connect(analyser)
  analyser.connect(context.destination)
  var bufferLength = analyser.frequencyBinCount
  var dataArray = new Float32Array(bufferLength)
  function update() {
    requestAnimationFrame(update)
    analyser.getFloatTimeDomainData(dataArray)
    player.value = dataArray
    player.dispatchEvent(new CustomEvent("input"))
}
  update()
}

navigator.mediaDevices.getUserMedia({ audio: true, video: false })
  .then(handleSuccess)
</script>
""")

I'm totally hacking away at this without having a clue what it is I'm doing LOL...it's a lot of fun though : )

[fonsp]

😊 no worries that's what javascript is designed for

[fonsp]

look forward to try it out soon!

[mthelm85]

I need to code a kill switch into it - right now it just runs indefinitely and there's no way to shut it off 🤣. It's streaming the data but it all goes so fast that I can't really tell if the values are useful...it's too fast for me to see if they are changing in an expected way when I speak into my mic....

[mthelm85]

Okay, I'm inching closer to something useful....try this one out:

@bind audio HTML("""
<audio id="player"></audio>
<button class="button" id="stopButton">Stop</button>
<script>
const player = document.getElementById('player')
const stop = document.getElementById('stopButton')

const handleSuccess = function(stream) {
  const context = new AudioContext({ sampleRate: 44100 })
  const analyser = context.createAnalyser()
  const source = context.createMediaStreamSource(stream)

  source.connect(analyser)
  analyser.connect(context.destination)
  var bufferLength = analyser.frequencyBinCount
  var dataArray = new Float32Array(bufferLength)
  const streamAudio = setInterval(function() {
    analyser.getFloatTimeDomainData(dataArray)
    player.value = dataArray
    player.dispatchEvent(new CustomEvent("input"))
  }, 1000)
  stop.addEventListener('click', () => {
    clearInterval(streamAudio)
  })
}

navigator.mediaDevices.getUserMedia({ audio: true, video: false })
  .then(handleSuccess)
</script>
<style>
.button {
  background-color: darkred;
  border: none;
  border-radius: 12px;
  color: white;
  padding: 15px 32px;
  text-align: center;
  text-decoration: none;
  display: inline-block;
  font-size: 16px;
  font-family: "Alegreya Sans", sans-serif;
  margin: 4px 2px;
  cursor: pointer;
}
</style>
""")

This will update every second and it's working as expected. Load Plots and SampledSignals and then in another cell do plot(domain(SampleBuf(Array(audio), 44100)), SampleBuf(Array(audio), 44100), legend=false).

[fonsp]

I don't have access, can you use gist.github.com isntead?

[mthelm85]

Sorry about that, try this link instead: https://drive.google.com/file/d/1F7qxBBojMl97ubSiuJ44EGVXht6TrJag/view?usp=sharing

It's a recording of my screen with audio so I don't think I can upload that to a gist.

[fonsp]

Haha I love it! Can you make it faster?

[fonsp]

I played around with createScriptProcessor instead of createAnalyser - it seems like the results are less jumpy but i also have no clue what i am doing

@bind audio HTML("""
<z id="player"></z>
<button class="button" id="stopButton">Stop</button>
<script>
const player = document.getElementById('player')
const stop = document.getElementById('stopButton')

const handleSuccess = function(stream) {
    const context = new AudioContext()
    const source = context.createMediaStreamSource(stream)
    const processor = context.createScriptProcessor(1024, 1, 1);

    source.connect(processor)
    processor.connect(context.destination)

    processor.onaudioprocess = function(e) {
        const data = e.inputBuffer.getChannelData(0)

        player.value = data
        player.dispatchEvent(new CustomEvent("input"))
        if(!document.body.contains(player)){
            processor.onaudioprocess = undefined
        }
    }

    stop.onclick = () => {processor.onaudioprocess = undefined}
}

navigator.mediaDevices.getUserMedia({ audio: true, video: false })
  .then(handleSuccess)
</script>
<style>
.button {
  background-color: darkred;
  border: none;
  border-radius: 12px;
  color: white;
  padding: 15px 32px;
  text-align: center;
  text-decoration: none;
  display: inline-block;
  font-size: 16px;
  font-family: "Alegreya Sans", sans-serif;
  margin: 4px 2px;
  cursor: pointer;
}
</style>
""")

[mthelm85]

Haha I love it! Can you make it faster?

Yes, if you just replace the 1000 with 0 in the setInterval function it goes to near real time. Also, it's best to set ylims=(-1,1) in the plot. I got out my guitar this time so that I don't sound like I'm making whale calls 😂

https://drive.google.com/file/d/1iVxtIjV7UK2LXF5Oifw7SapWYVu1v9cq/view?usp=sharing

The reason I chose the AnalyserNode is that the Mozilla docs state specifically that it's for data analysis/visualization.

[mthelm85]

Here's the full code for what I did in the video linked above:

@bind audio HTML("""
<audio id="player"></audio>
<button class="button" id="stopButton">Stop</button>
<script>
const player = document.getElementById('player')
const stop = document.getElementById('stopButton')

const handleSuccess = function(stream) {
  const context = new AudioContext({ sampleRate: 44100 })
  const analyser = context.createAnalyser()
  const source = context.createMediaStreamSource(stream)

  source.connect(analyser)
  analyser.connect(context.destination)
  const bufferLength = analyser.frequencyBinCount
  let dataArray = new Float32Array(bufferLength)
  const streamAudio = setInterval(function() {
    analyser.getFloatTimeDomainData(dataArray)
    player.value = dataArray
    player.dispatchEvent(new CustomEvent("input"))
  }, 0)
  stop.onclick = () => { clearInterval(streamAudio) }
}

navigator.mediaDevices.getUserMedia({ audio: true, video: false })
  .then(handleSuccess)
</script>
<style>
.button {
  background-color: darkred;
  border: none;
  border-radius: 12px;
  color: white;
  padding: 15px 32px;
  text-align: center;
  text-decoration: none;
  display: inline-block;
  font-size: 16px;
  font-family: "Alegreya Sans", sans-serif;
  margin: 4px 2px;
  cursor: pointer;
}
</style>
""")

using Plots, SampledSignals

plot(domain(SampleBuf(Array(audio), 44100)), SampleBuf(Array(audio), 44100), legend=false, ylims=(-1,1))

[fonsp]

Thank you for the tiny concert, that was wonderful ❤

[mthelm85]

haha, glad you liked it! ; )

[elihugarret]

Hi! 😄 Im the author of the tweet. Was experimenting some web audio synths and Pluto but I used almost the same approach of @mthelm85, also used the canvas to draw the waveform at the same time.

Been experimenting with @mthelm85 code and it performs better of what I did in the tweet. I just made a few changes, like disconnecting the analyser to avoid feedback when not using headphones, and replaced setInterval with the requestAnimationFrame function that seems more appropriate for this use case. Also wrote one notebook with MIDI input here: https://gist.github.com/elihugarret/fc3de3600c972a64246aa3e3efb96611

@bind audio HTML("""
<audio id="player"></audio>
<button class="button" id="stopButton">Stop</button>
<script>
  const player = document.getElementById('player');
  const stop = document.getElementById('stopButton');

  const handleSuccess = function(stream) {
    const context = new AudioContext({ sampleRate: 44100 });
    const analyser = context.createAnalyser();
    const source = context.createMediaStreamSource(stream);

    source.connect(analyser);

    const bufferLength = analyser.frequencyBinCount;

    let dataArray = new Float32Array(bufferLength);
    let animFrame;

    const streamAudio = () => {
      animFrame = requestAnimationFrame(streamAudio);
      analyser.getFloatTimeDomainData(dataArray);
      player.value = dataArray;
      player.dispatchEvent(new CustomEvent("input"));
    }

    streamAudio();

    stop.onclick = e => {
      source.disconnect(analyser);
      cancelAnimationFrame(animFrame);
    }
  }

  navigator.mediaDevices.getUserMedia({ audio: true, video: false })
    .then(handleSuccess)
</script>
<style>
.button {
  background-color: darkred;
  border: none;
  border-radius: 12px;
  color: white;
  padding: 15px 32px;
  text-align: center;
  text-decoration: none;
  display: inline-block;
  font-size: 16px;
  font-family: "Alegreya Sans", sans-serif;
  margin: 4px 2px;
  cursor: pointer;
}
</style>
""")

[mthelm85]

Nice! This is coming full circle 😃

[fonsp]

How is it going over here?

[mthelm85]

Unfortunately, I've not worked on it anymore. I'll actually have a bit of free time in the coming weeks though and I'd love to help out. Do you have a more clear idea yet of what you would like a finished product to look like for this microphone input?

[fonsp]

I don't :)

[mthelm85]

How about I attempt to add this and submit a PR and then hopefully users will begin chiming in on what would make it useful...??

[fonsp]

okay!

[mthelm85]

PR #54