RetellAI / retell-client-js-sdk

10 stars 9 forks source link

Audio stutter when trying to playback raw audio data #15

Open AnasMations opened 2 months ago

AnasMations commented 2 months ago

Hello, we are adding a facial animation model to Retell's voice agent so we need to pass in the raw audio data stream to our model as PCM16. We first convert that from Float32 to Uint8. The audio doesn't work and it starts stuttering badly, It used to work perfectly with the old websockets based implementation.

This is how it sounds 👇 https://github.com/user-attachments/assets/4fdb7359-9d1f-4c47-accc-2ec54df489d0

Thanks!

gudj0 commented 2 months ago

Any update on this from the Retell team?

Our client is expecting uint8. We added the convertFloat32ToUnsigned8 from your docs: https://docs.retellai.com/knowledge/audio-basics

Still experiencing massive stutter.

Example snippet from our code:

    retellWebClient.on('audio', (audio: Float32Array) => {
        const processedAudio = convertFloat32ToUnsigned8(audio);
        simliClientRef.current.sendAudioData(processedAudio);
      } 
    });
levihatchmatter commented 1 month ago

I'm not sure if this will help with your particular issue but I ran into something similar. You could try wrapping the handler with useCallback.

const sendAudio = useCallback((audio: Float32Array) => {
        const processedAudio = convertFloat32ToUnsigned8(audio);
        simliClientRef.current.sendAudioData(processedAudio);
      }, [] );

retellWebClient.on('audio', sendAudio);