How to send real time audio streams to web socket?

[VincentLu91]

Hi,

From my understanding, the useReactMediaRecorder allows the Reactjs app to record microphone + screen audio together:

import { useReactMediaRecorder } from "react-media-recorder";

const InternalRecording = () => {
  const { status, startRecording, stopRecording, mediaBlobUrl } =
    useReactMediaRecorder({ audio: true });
  return (
    <div>
      <p>{status}</p>
      <button onClick={startRecording}>Start Recording</button>
      <button onClick={stopRecording}>Stop Recording</button>
      <video src={mediaBlobUrl} controls autoPlay loop />
    </div>
  );
};
export default InternalRecording;

I've experienced this myself when I was playing a YouTube video while speaking on my computer microphone as the web app is recording.

I want to be able to send this overall real time audio stream (blob) to the web socket during recording.

Here's how I tried:

socketRef.current = await new WebSocket(
        "<API URL>"
      );

      const texts = {};
// further down
recorder = RecordRTC(previewAudioStream, {
      type: "audio",
      mimeType: "audio/webm;codecs=pcm", 
      recorderType: StereoAudioRecorder, 
      timeSlice: 250, // set 250 ms intervals of data that sends to AAI
      //sampleRate: 16000,
      desiredSampRate: 16000,
      numberOfAudioChannels: 2, 
      bufferSize: 4096,
      //audioBitsPerSecond: 128000,
      ondataavailable: (blob) => {
        console.log("trying to read blob: ", blob);
        const reader = new FileReader();
        reader.onload = () => {
          const base64data = reader.result;
          console.log("socket is: ", socketRef.current);
          console.log("base64data is: ", base64data);
          // audio data must be sent as a base64 encoded string
          if (socketRef.current) {
            socketRef.current.send(
              JSON.stringify({
                audio_data: base64data.split("base64,")[1],
              })
            );
          }
        };
        reader.readAsDataURL(blob);
      },
    });

    recorder.startRecording();

I think I'm doing this all wrong as it records just from the microphone. Is there a way to properly transmit the overall audio stream (microphone + screen audio) to the web socket? I want to be able to process the overall audio using an API.

[VahidNZ]

I'm having the same problem. Actually, I have a pure Javascript code that takes stream audio and encodes it to a base64 string before sending it to a webSocket, however I can't accomplish this with React js; any suggestions?

[Guuri11]

@VahidNZ I could convert it to base64 & send it to ws. I'll share my code asap. I'm don't have my laptop now

[Guuri11]

@VahidNZ here is the code

/** CONVERT BLOB TO BASE64 */
const blobToBase64 = (blob: any) => new Promise((resolve, reject) => {
    const reader = new FileReader();

    reader.readAsDataURL(blob);
    reader.onload = () => resolve(reader.result);
    reader.onerror = (error) => reject(error);
  });
  const convertBlobToBase64 = async (blob: any) => {
    // get blob from http url
    const response = await fetch(blob);
        // transform to blob
    const audioMessageBlob = await response.blob();
    return blobToBase64(audioMessageBlob);
  };
/** CONVERT BLOB TO BASE64 END */

  // This is called when I stop recording my audio, I process the blob & then send it to ws
  const sendValue = async () => {
    if (stompClient) {
      stopRecording();
      convertBlobToBase64(mediaBlobUrl).then((sourceAudio) => {
        const chatMessage = {
          sentBy: user.backendUser?.id,
          source: sourceAudio,
          status: 'MESSAGE',
        };
        stompClient.send('/app/message', {}, JSON.stringify(chatMessage));
      });
    }
  };

[deep-inexture]

Hi,

From my understanding, the useReactMediaRecorder allows the Reactjs app to record microphone + screen audio together:

import { useReactMediaRecorder } from "react-media-recorder";

const InternalRecording = () => {
  const { status, startRecording, stopRecording, mediaBlobUrl } =
    useReactMediaRecorder({ audio: true });
  return (
    <div>
      <p>{status}</p>
      <button onClick={startRecording}>Start Recording</button>
      <button onClick={stopRecording}>Stop Recording</button>
      <video src={mediaBlobUrl} controls autoPlay loop />
    </div>
  );
};
export default InternalRecording;

I've experienced this myself when I was playing a YouTube video while speaking on my computer microphone as the web app is recording.

I want to be able to send this overall real time audio stream (blob) to the web socket during recording.

Here's how I tried:

socketRef.current = await new WebSocket(
        "<API URL>"
      );

      const texts = {};
// further down
recorder = RecordRTC(previewAudioStream, {
      type: "audio",
      mimeType: "audio/webm;codecs=pcm", 
      recorderType: StereoAudioRecorder, 
      timeSlice: 250, // set 250 ms intervals of data that sends to AAI
      //sampleRate: 16000,
      desiredSampRate: 16000,
      numberOfAudioChannels: 2, 
      bufferSize: 4096,
      //audioBitsPerSecond: 128000,
      ondataavailable: (blob) => {
        console.log("trying to read blob: ", blob);
        const reader = new FileReader();
        reader.onload = () => {
          const base64data = reader.result;
          console.log("socket is: ", socketRef.current);
          console.log("base64data is: ", base64data);
          // audio data must be sent as a base64 encoded string
          if (socketRef.current) {
            socketRef.current.send(
              JSON.stringify({
                audio_data: base64data.split("base64,")[1],
              })
            );
          }
        };
        reader.readAsDataURL(blob);
      },
    });

    recorder.startRecording();

I think I'm doing this all wrong as it records just from the microphone. Is there a way to properly transmit the overall audio stream (microphone + screen audio) to the web socket? I want to be able to process the overall audio using an API.

Hey, I am looking for something similar, but I am unable to understand how can I implement. Can you please explain?

[vijulshah]

I did the same for videoStream. I believe the same solution can work for the audio stream. So, here is my solution:

let mediaRecorder = new MediaRecorder(localStream, options)
mediaRecorder.ondataavailable = handleDataAvailable
mediaRecorder.start(1000)

async function handleDataAvailable(event) {
        if(event.data && event.data.size > 0) {
            let blobBuffer = new Blob([event.data],{type:'video/webm'})
            let rawbuffer = await blobBuffer.arrayBuffer()
            let buffer = new Uint8Array(rawbuffer)
            webSocket.postMessage(buffer)
        }
    }

Then you will get data as bytes in the socket server.

[prdip]

I did the same for videoStream. I believe the same solution can work for the audio stream. So, here is my solution:

let mediaRecorder = new MediaRecorder(localStream, options)
mediaRecorder.ondataavailable = handleDataAvailable
mediaRecorder.start(1000)

async function handleDataAvailable(event) {
        if(event.data && event.data.size > 0) {
            let blobBuffer = new Blob([event.data],{type:'video/webm'})
            let rawbuffer = await blobBuffer.arrayBuffer()
            let buffer = new Uint8Array(rawbuffer)
            webSocket.postMessage(buffer)
        }
    }

Then you will get data as bytes in the socket server.

Can you share your videoStream realtime websocket upload code here ?