charisma-ai / charisma-react

Charisma.ai chat component for React
MIT License
4 stars 2 forks source link

add audio/character audio to test project #33

Open John-A-J opened 5 months ago

John-A-J commented 5 months ago

maybe follow this up with a ticket to stop window needing to be monkey patched in the janky way?

John-A-J commented 4 months ago

Requirements: A solution that works on edge, chrome, firefox, safari x iOS, windows, android, macOS

Can't use WebCodecs API as it's not supported on Firefox.

Using AudioWorkletNode but Safari has potential bugs, and requires either different implementations for safari/chrome, or very particular ordered interactions.

Use MediaRecorder? not supported on IE, Opera Mini, Android Browser, or Firefox for Android. I think that's fine.

scenarios of event order to consider testing background audio, character audio, mic input, back to same background audio mic input, background audio, character audio, mic input, back to same background audio character audio, mic input, character audio mic input, character audio

John-A-J commented 4 months ago

We should definitely use https://github.com/chrisguttandin/standardized-audio-context

relevant warnings in its docs

⚠️ AudioContext - Setting the sampleRate is not supported in Safari so far.

⚠️ audioWorklet - The AudioWorklet is accessible as a property of an AudioContext or OfflineAudioContext. It uses the ScriptProcessorNode internally to create an AudioWorkletProcessor in Safari. This means it will only provide the performance improvements that you would normally expect from using an AudioWorklet in Chrome, Edge and Firefox.

⚠️ createBuffer() / AudioBuffer - Safari does not support AudioBuffers with a sampleRate below 22050 Hz.

⚠️ createMediaStreamSource() / MediaStreamAudioSourceNode - Safari outputs silence if the MediaStreamAudioSourceNode is disconnected for about two seconds.