Attendee gets `sendingAudioFailed` occasionally when joining a meeting with video turned on

evangeline commented 1 year ago

What happened and what did you expect to happen?

We very often see attendees not being heard when they join a meeting with both audio and video turned on. However, this never seems to happen if they join a meeting with just audio turned on. When trying to reproduce the issue locally, we see sendingAudioFailed logged for these attendees.

For context, we pass in mediaStreams instead of device/device ids into startAudioInput and startVideoInput. I have verified that the audio track is still live when this bug happens. We have looked through our logs repeatedly over the last few weeks and can't find anything that stands out.

Have you reviewed our existing documentation?

[X] GitHub issues
[X] FAQs page
[X] README page
[X] Feature-specific guides

Reproduction steps

Difficult to create a repro with your demo app because our app is set up very differently, added logs below for debugging.

Amazon Chime SDK for JavaScript version

3.14.1

What browsers are you seeing the problem on?

Chrome

Browser version

114.0.5735.198

Meeting and Attendee ID Information.

Audio sends successfully (video turned off): Meeting ID: 03e6c658-b43c-4f48-b05c-0250efdf2713 Attendee ID: 8f206d4d-076e-6253-0edd-af487d717e19

Failed to send audio (video turned on): Meeting ID: a84da418-15ae-4057-9c1b-7ba203e52713 Attendee ID: a5f4268f-096c-27ac-9d8c-235a9162ff8f

Browser console logs

Logs for both meetings attached:

chime.txt

xuesichao commented 1 year ago

Hi @evangeline , since you are passing your own media stream into SDK, we can't reproduce this issue on our side. Could you explain

What kind of audio stream they were sending? e.g. source, codec, …
Could you please provide the client/browser-side log for the meeting where the audio failed to send in your AWS support ticket? The log attached in the ticket is for the meeting where the audio was sent successfully."

evangeline commented 1 year ago

Hi, they are codec streams.

I previously lumped the logs from both meetings into 1 file but have now split it into two separate files: audio_sent_successfully.txt failed_to_send_audio.txt

To clarify, whenever the audio fails to send, calling startAudioInput again with the same stream always fixes the issue. This bug is also inconsistent, so we suspect there's a race but are unable to determine where.

evangeline commented 1 year ago

When I tried changing my code from:

// simplified code
const mediaStream = new MediaStream([audioTrack]);
await this.audioVideo?.startAudioInput(mediaStream);

to

const audioInputDevices = await this.audioVideo.listAudioInputDevices(true);
await this.audioVideo?.startAudioInput(audioInputDevices[0]);

my audio always sends successfully, regardless of whether my video is turned on or off.

For the first code snippet, I have double checked that the track has not been accidentally ended. It also recovers when we call startAudioInput(mediaStream) again. It's very puzzling that audio sends correctly in most situations, except when joining a meeting with video is turned on.

It also works if I do something hacky like:

        setTimeout(async () => {
          await this.audioVideo?.startAudioInput(mediaStream);
        }, 1000)

I would like to avoid workarounds like this though.

For additional context, we always clone our audio tracks and put them in a new media stream if we call startAudioInput. This is because Chime may end our tracks, and we prefer to maintain control over when our tracks start and end.

evangeline commented 1 year ago

Hi! I managed to repro this bug with your browser demo, please see the PR here: https://github.com/aws/amazon-chime-sdk-js/pull/2712

devalevenkatesh commented 1 year ago

@evangeline thanks for the repro, will have to check the repro in the browser demo. But before calling startAudioInput its a required pre-req to call listAudioInputDevices API. Per your last comment, you mentioned that you do not run into the same issue when using listAudioInputDevices, could you confirm again?

We document this in README under Device section: https://github.com/aws/amazon-chime-sdk-js#device

I resolve this issue as we have a support ticket for the same. Will post the updates on the investigation as progressed in the support ticket.

devalevenkatesh commented 1 year ago

Posting here as it may help other builders. But will resolve for further question you can use the support ticket.

Log dive deep for failed_to_send_audio.txt:

The application lists the audio input devices:

Logger.js:11 04:33:50:144 2023-07-17T04:33:50.143Z [INFO] [Chime] - API/DefaultDeviceController/listAudioInputDevices true -> [{"deviceId":"default","kind":"audioinput","label":"Default - WH-1000XM3 (Bluetooth)","groupId":"5c723f6ac2d6ff3b58f1c01cdec2bb1075e1d8aa023ba408fd9ea0cbd5e89fef"},{"deviceId":"556324de86f83484c47fb4d37254539d9a0ba805b228626fbc5c8469171baea2","kind":"audioinput","label":"MacBook Pro Microphone (Built-in)","groupId":"3206834b0d12747bed12325f54a1063a91ed7efbb4758dd1cbdb790c275f09eb"},{"deviceId":"0eda69f310f4f6e3c871ec9e1ffad3466659cc8b0677858ffa45cde85d33604b","kind":"audioinput","label":"WH-1000XM3 (Bluetooth)","groupId":"5c723f6ac2d6ff3b58f1c01cdec2bb1075e1d8aa023ba408fd9ea0cbd5e89fef"},{"deviceId":"1688d839499a930fc5f46288fb6d13202c1fa762bcae37f6dd16fe7b6fbce98c","kind":"audioinput","label":"ZoomAudioDevice (Virtual)","groupId":"c4c1d51a3510928422fbb748985af1863fa0dfb699b6ab792ac776fdf48c382f"}]
Logger.js:11 04:33:50:144 2023-07-17T04:33:50.144Z [INFO] [Chime] - API/DefaultAudioVideoFacade/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f/listAudioInputDevices true -> [{"deviceId":"default","kind":"audioinput","label":"Default - WH-1000XM3 (Bluetooth)","groupId":"5c723f6ac2d6ff3b58f1c01cdec2bb1075e1d8aa023ba408fd9ea0cbd5e89fef"},{"deviceId":"556324de86f83484c47fb4d37254539d9a0ba805b228626fbc5c8469171baea2","kind":"audioinput","label":"MacBook Pro Microphone (Built-in)","groupId":"3206834b0d12747bed12325f54a1063a91ed7efbb4758dd1cbdb790c275f09eb"},{"deviceId":"0eda69f310f4f6e3c871ec9e1ffad3466659cc8b0677858ffa45cde85d33604b","kind":"audioinput","label":"WH-1000XM3 (Bluetooth)","groupId":"5c723f6ac2d6ff3b58f1c01cdec2bb1075e1d8aa023ba408fd9ea0cbd5e89fef"},{"deviceId":"1688d839499a930fc5f46288fb6d13202c1fa762bcae37f6dd16fe7b6fbce98c","kind":"audioinput","label":"ZoomAudioDevice (Virtual)","groupId":"c4c1d51a3510928422fbb748985af1863fa0dfb699b6ab792ac776fdf48c382f"}]

Meeting session start is triggered:

// meetingSession.audioVideo.start()
Logger.js:11 04:33:50:153 2023-07-17T04:33:50.153Z [INFO] [Chime] - transitioning from NotConnected to Connecting with Connect
.
.
Logger.js:11 04:33:50:153 2023-07-17T04:33:50.153Z [INFO] [Chime] - running task AudioVideoStart/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f

JS SDK selects empty audio device internally (null) as startAudioInput is not called before starting the meeting:

Logger.js:11 04:33:50:158 2023-07-17T04:33:50.158Z [INFO] [Chime] - running task ReceiveAudioInputTask
Logger.js:11 04:33:50:158 2023-07-17T04:33:50.158Z [INFO] [Chime] - No audio device chosen, creating empty audio device

Next, empty device is successfully chosen:

Logger.js:11 04:33:50:159 2023-07-17T04:33:50.159Z [INFO] [Chime] - Choosing intrinsic audio input device null
Logger.js:11 04:33:50:159 2023-07-17T04:33:50.159Z [INFO] [Chime] - requesting new audio device with constraint null
Logger.js:11 04:33:50:159 2023-07-17T04:33:50.159Z [INFO] [Chime] - got audio device for constraints null
Logger.js:11 04:33:50:160 2023-07-17T04:33:50.160Z [INFO] [Chime] - API/DefaultDeviceController/startAudioInputDevice null -> "success"

A video is attempted to start using startLocalVideoTile, along with audio with a media stream, both succeed, but start video requires SDP re-negotiation which deffers the media stream update until the initial AudioVideoStart (meetingSession.audioVideo.start()) completes:

Logger.js:11 04:33:50:163 2023-07-17T04:33:50.163Z [INFO] [Chime] - API/DefaultAudioVideoFacade/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f/startVideoInput {}
Logger.js:11 04:33:50:164 2023-07-17T04:33:50.164Z [INFO] [Chime] - API/DefaultAudioVideoFacade/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f/startAudioInput {}
Logger.js:11 04:33:50:165 2023-07-17T04:33:50.165Z [INFO] [Chime] - Update request requires resubscribe
Logger.js:11 04:33:50:165 2023-07-17T04:33:50.165Z [INFO] [Chime] - deferring transition from Connecting with Update
Logger.js:11 04:33:50:165 2023-07-17T04:33:50.165Z [INFO] [Chime] - API/DefaultAudioVideoFacade/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f/startLocalVideoTile null -> 1
Logger.js:11 04:33:50:165 2023-07-17T04:33:50.165Z [INFO] [Chime] - requesting new video device with constraint {"video":{"streamId":"3924b447-8319-4ed7-bf21-ffe1e3a3a624"}}
Logger.js:11 04:33:50:165 2023-07-17T04:33:50.165Z [INFO] [Chime] - using media stream 3924b447-8319-4ed7-bf21-ffe1e3a3a624 for video device
Logger.js:11 04:33:50:165 2023-07-17T04:33:50.165Z [INFO] [Chime] - got video device for constraints {"video":{"streamId":"3924b447-8319-4ed7-bf21-ffe1e3a3a624"}}
Logger.js:11 04:33:50:166 2023-07-17T04:33:50.166Z [INFO] [Chime] - API/DefaultDeviceController/startVideoInputDevice {}

Now at this point a call to startAudioInput is seen which succeeds:

Logger.js:11 04:33:50:166 2023-07-17T04:33:50.166Z [INFO] [Chime] - Choosing intrinsic audio input device [object MediaStream]
Logger.js:11 04:33:50:166 2023-07-17T04:33:50.166Z [INFO] [Chime] - requesting new audio device with constraint {"audio":{"streamId":"fde140ff-4f5b-4a53-9809-86b0b13a4c6b"}}
Logger.js:11 04:33:50:166 2023-07-17T04:33:50.166Z [INFO] [Chime] - using media stream fde140ff-4f5b-4a53-9809-86b0b13a4c6b for audio device
Logger.js:11 04:33:50:166 2023-07-17T04:33:50.166Z [INFO] [Chime] - got audio device for constraints {"audio":{"streamId":"fde140ff-4f5b-4a53-9809-86b0b13a4c6b"}}
Logger.js:11 04:33:50:166 2023-07-17T04:33:50.166Z [INFO] [Chime] - ReceiveAudioInputTask took 8 ms
Logger.js:11 04:33:50:166 2023-07-17T04:33:50.166Z [INFO] [Chime] - API/DefaultDeviceController/startAudioInputDevice {} -> "success"
Logger.js:11 04:33:50:166 2023-07-17T04:33:50.166Z [INFO] [Chime] - Update request requires resubscribe
Logger.js:11 04:33:50:166 2023-07-17T04:33:50.166Z [INFO] [Chime] - deferring transition from Connecting with Update

Technically audio has not yet reached the Chime media servers, but the meetingSession.audioVideo.start() now completes:

Logger.js:11 04:33:50:608 2023-07-17T04:33:50.608Z [INFO] [Chime] - timeout task AudioVideoStart/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f/Timeout15000ms completed
Logger.js:11 04:33:50:608 2023-07-17T04:33:50.608Z [INFO] [Chime] - AudioVideoStart/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f/Timeout15000ms took 452 ms
Logger.js:11 04:33:50:608 2023-07-17T04:33:50.608Z [INFO] [Chime] - serial group task AudioVideoStart/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f completed subtask AudioVideoStart/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f/Timeout15000ms
Logger.js:11 04:33:50:608 2023-07-17T04:33:50.608Z [INFO] [Chime] - serial group task AudioVideoStart/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f completed
Logger.js:11 04:33:50:608 2023-07-17T04:33:50.608Z [INFO] [Chime] - AudioVideoStart/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f took 455 ms
Logger.js:11 04:33:50:608 2023-07-17T04:33:50.608Z [INFO] [Chime] - transitioning from Connecting to Connected with FinishConnecting

Now, the deferred action to update the audio video selection is executed:

Logger.js:11 04:33:50:610 2023-07-17T04:33:50.610Z [INFO] [Chime] - performing deferred action Update
Logger.js:11 04:33:50:610 2023-07-17T04:33:50.610Z [INFO] [Chime] - transitioning from Connected to Updating with Update
Logger.js:11 04:33:50:610 2023-07-17T04:33:50.610Z [INFO] [Chime] - running task AudioVideoUpdate/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f

As the AudioVideoUpdate is in progress, another startAudioInput is called and audio-video are updated and attendee presence is received:

Logger.js:11 04:33:50:612 2023-07-17T04:33:50.612Z [INFO] [Chime] - API/DefaultAudioVideoFacade/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f/startAudioInput {}
Logger.js:11 04:33:50:612 2023-07-17T04:33:50.612Z [INFO] [Chime] - Choosing intrinsic audio input device [object MediaStream]
Logger.js:11 04:33:50:612 2023-07-17T04:33:50.612Z [INFO] [Chime] - requesting new audio device with constraint {"audio":{"streamId":"9edd54ec-ea66-4723-80bc-895bb4d848d0"}}
Logger.js:11 04:33:50:612 2023-07-17T04:33:50.612Z [INFO] [Chime] - using media stream 9edd54ec-ea66-4723-80bc-895bb4d848d0 for audio device
Logger.js:11 04:33:50:612 2023-07-17T04:33:50.612Z [INFO] [Chime] - got audio device for constraints {"audio":{"streamId":"9edd54ec-ea66-4723-80bc-895bb4d848d0"}}
Logger.js:11 04:33:50:613 2023-07-17T04:33:50.613Z [INFO] [Chime] - API/DefaultDeviceController/startAudioInputDevice {} -> "success"
Logger.js:11 04:33:50:613 2023-07-17T04:33:50.613Z [INFO] [Chime] - Receive an audio input change event
..
.
.
Logger.js:11 04:33:50:614 2023-07-17T04:33:50.614Z [INFO] [Chime] - attaching audio track to peer connection
Logger.js:11 04:33:50:614 2023-07-17T04:33:50.614Z [INFO] [Chime] - Local audio input is updated
Logger.js:11 04:33:50:615 2023-07-17T04:33:50.615Z [INFO] [Chime] - attaching video track to peer connection
Logger.js:11 04:33:50:615 2023-07-17T04:33:50.615Z [INFO] [Chime] - peer connection negotiation is needed
.
.
.
Logger.js:11 04:33:50:638 2023-07-17T04:33:50.638Z [INFO] [Chime] - AudioVideoUpdate/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f took 28 ms
Logger.js:11 04:33:50:638 2023-07-17T04:33:50.638Z [INFO] [Chime] - transitioning from Updating to Connected with FinishUpdating
Logger.js:11 04:33:50:638 2023-07-17T04:33:50.638Z [INFO] [Chime] - Resuming index ingestion with pending index
Logger.js:11 04:33:50:638 2023-07-17T04:33:50.638Z [INFO] [Chime] - should resubscribe: false (downlink: false uplink: false)
Logger.js:11 04:33:50:639 2023-07-17T04:33:50.639Z [INFO] [Chime] - updated audio-video session
.
.
Logger.js:11 04:33:50:831 2023-07-17T04:33:50.831Z [INFO] [Chime] - attendeePresenceReceived: a5f4268f-096c-27ac-9d8c-235a9162ff8f

Since, the audio was not actually been processed for some time after the meetingSession.audioVideo.start() completed (and it only ever succeeded with the deferred update), the connection health policy within JS SDK says:

Logger.js:11 04:33:54:159 2023-07-17T04:33:54.159Z [WARN] [Chime] - Sending Audio is unhealthy for 2 seconds consecutively.
Logger.js:11 04:33:54:159 2023-07-17T04:33:54.159Z [INFO] [Chime] - Sending Audio Health value is now 0

At this point, the last audio video update is successful so audio should be flowing fine + attendee presence is received, no further startAudioInput is needed, but it seems its called again. The connection health policy on its own registers healthy audio now:

Logger.js:11 04:33:54:160 2023-07-17T04:33:54.160Z [INFO] [Chime] - API/DefaultAudioVideoFacade/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f/startAudioInput {}
Logger.js:11 04:33:54:160 2023-07-17T04:33:54.160Z [INFO] [Chime] - Choosing intrinsic audio input device [object MediaStream]
Logger.js:11 04:33:54:160 2023-07-17T04:33:54.160Z [INFO] [Chime] - requesting new audio device with constraint {"audio":{"streamId":"77bbf616-91e9-442a-9398-97a5725cdf8a"}}
Logger.js:11 04:33:54:161 2023-07-17T04:33:54.161Z [INFO] [Chime] - using media stream 77bbf616-91e9-442a-9398-97a5725cdf8a for audio device
Logger.js:11 04:33:54:161 2023-07-17T04:33:54.161Z [INFO] [Chime] - got audio device for constraints {"audio":{"streamId":"77bbf616-91e9-442a-9398-97a5725cdf8a"}}
Logger.js:11 04:33:54:161 2023-07-17T04:33:54.161Z [INFO] [Chime] - API/DefaultDeviceController/startAudioInputDevice {} -> "success"
Logger.js:11 04:33:54:161 2023-07-17T04:33:54.161Z [INFO] [Chime] - Receive an audio input change event
Logger.js:11 04:33:54:161 2023-07-17T04:33:54.161Z [INFO] [Chime] - API/DefaultAudioVideoFacade/a84da418-15ae-4057-9c1b-7ba203e52713/a5f4268f-096c-27ac-9d8c-235a9162ff8f/realtimeUnmuteLocalAudio
Logger.js:11 04:33:54:161 2023-07-17T04:33:54.161Z [INFO] [Chime] - Local audio input is updated
Logger.js:11 04:33:55:158 2023-07-17T04:33:55.158Z [INFO] [Chime] - Sending Audio Health value is now 1

Log dive deep for audio_sent_successfully.txt:

Everything is same to above failed case logs until empty device is chosen:

Logger.js:11 04:29:55:841 2023-07-17T04:29:55.841Z [INFO] [Chime] - No audio device chosen, creating empty audio device
Logger.js:11 04:29:55:841 2023-07-17T04:29:55.841Z [INFO] [Chime] - running task Signaling/Timeout15000ms
Logger.js:11 04:29:55:841 2023-07-17T04:29:55.841Z [INFO] [Chime] - running task Signaling/Timeout15000ms/OpenSignalingConnectionTask
Logger.js:11 04:29:55:841 2023-07-17T04:29:55.841Z [INFO] [Chime] - adding connection request to queue: wss://signal.m1.as1.app.chime.aws/control/03e6c658-b43c-4f48-b05c-0250efdf2713?X-Chime-Control-Protocol-Version=3&X-Amzn-Chime-Send-Close-On-Error=1
Logger.js:11 04:29:55:841 2023-07-17T04:29:55.841Z [INFO] [Chime] - no existing signaling client connection needs closing
Logger.js:11 04:29:55:841 2023-07-17T04:29:55.841Z [INFO] [Chime] - opening connection to wss://signal.m1.as1.app.chime.aws/control/03e6c658-b43c-4f48-b05c-0250efdf2713?X-Chime-Control-Protocol-Version=3&X-Amzn-Chime-Send-Close-On-Error=1
Logger.js:11 04:29:55:842 2023-07-17T04:29:55.842Z [INFO] [Chime] - notifying event: WebSocketConnecting
Logger.js:11 04:29:55:842 2023-07-17T04:29:55.842Z [INFO] [Chime] - Choosing intrinsic audio input device null
Logger.js:11 04:29:55:843 2023-07-17T04:29:55.843Z [INFO] [Chime] - requesting new audio device with constraint null
Logger.js:11 04:29:55:845 2023-07-17T04:29:55.845Z [INFO] [Chime] - got audio device for constraints null
Logger.js:11 04:29:55:845 2023-07-17T04:29:55.845Z [INFO] [Chime] - API/DefaultDeviceController/startAudioInputDevice null -> "success"
Logger.js:11 04:29:55:848 2023-07-17T04:29:55.848Z [INFO] [Chime] - API/DefaultAudioVideoFacade/03e6c658-b43c-4f48-b05c-0250efdf2713/8f206d4d-076e-6253-0edd-af487d717e19/startAudioInput {}
Logger.js:11 04:29:55:849 2023-07-17T04:29:55.849Z [INFO] [Chime] - Choosing intrinsic audio input device [object MediaStream]
Logger.js:11 04:29:55:849 2023-07-17T04:29:55.849Z [INFO] [Chime] - requesting new audio device with constraint {"audio":{"streamId":"17a58f9c-265e-4dc6-bdbe-f0ac31177166"}}
Logger.js:11 04:29:55:849 2023-07-17T04:29:55.849Z [INFO] [Chime] - using media stream 17a58f9c-265e-4dc6-bdbe-f0ac31177166 for audio device
Logger.js:11 04:29:55:850 2023-07-17T04:29:55.850Z [INFO] [Chime] - got audio device for constraints {"audio":{"streamId":"17a58f9c-265e-4dc6-bdbe-f0ac31177166"}}
Logger.js:11 04:29:55:850 2023-07-17T04:29:55.850Z [INFO] [Chime] - ReceiveAudioInputTask took 10 ms
Logger.js:11 04:29:55:850 2023-07-17T04:29:55.850Z [INFO] [Chime] - API/DefaultDeviceController/startAudioInputDevice {} -> "success"

But, since startVideoInput is not called, there is no requirement for AudioVideoUpdate (SDP renegotiation) and thats the culprit. The update defers the audio input selection finalizing and hence never completes within the meetingSession.audioVideo.start() resulting into sendingAudioFailed.

Root cause

startVideoInput is called before meetingSession.audioVideo.start() completes leading to a deferred update requiring an SDP negotiation (a step in WebRTC connection setup with Chime media servers). Thus, the startAudioInput is delayed until the next update can be applied which results into sendingAudioFailed as there is no audio yet passing through Chime media servers and thus no attendeePresenceReceived yet.
The startVideoInput is actually a result of calling startLocalVideoTile.
Calling startAudioInput multiple times eventually helps as the session is then set with a valid media stream audio device and everything works fine.
startAudioInput is called after AudioVideoStart that means, startAudioInput API is called after meetingSession.audioVideo.start() API. Same for startLocalVideoTile.

Solution

Call startAudioInput API and await it before calling meetingSession.audioVideo.start() API.
Call startLocalVideoTile post meetingSession.audioVideo.start() completes. Thus, you have to implement the audioVideoDidStart observer and in there once you receive an update in that observer call startLocalVideoTile to start the local video.

evangeline commented 1 year ago

@devalevenkatesh Thanks for the detailed explanation, we've followed your suggestions and are still observing audio failing to send when joining a meeting with video turned on. In other words, the user could not be heard by other attendees, even though he is unmuted, when he joins the meeting with both audio and video unmuted.

What I've changed:

We now list and start the default devices first before calling meetingSession.audioVideo.start() to avoid races. Creating of audio and video streams is handled elsewhere in the app and there is no guarantee that they would be received before a meeting is started.
We keep audio and video muted by calling realtimeMuteLocalAudio() and stopLocalVideoTile() to avoid publishing with the default device.
We bind the audio element with bindAudioElement().
We start the meeting with meetingSession.audioVideo.start().
After we receive the event audioVideoDidStart, we then call startAudioInput and startVideoInput again when the streams are received. (As we do pre-processing on audio/video tracks, we will call the corresponding APIs whenever the track is changed)
If the user has already unmuted themselves, we also call realtimeUnmuteLocalAudio() or startLocalVideoTile().

I have attached another log that repro-ed the issue here. There were no empty audio devices created in this repro: failed_to_send_audio_20_Jul.txt

I wonder if the cause of this is calling startAudioInput, realtimeUnmuteLocalAudio, startVideoInput, startLocalVideoTile in quick succession. We do await each call, but this bug seems to only happen when joining a meeting with both audio and video unmuted. We do have an unusual use case where we're passing in media streams programmatically instead of waiting for a user action (eg a physical click to unmute).

I have also tried calling realtimeUnmuteLocalAudio and startLocalVideoTile after receiving audioInputSelected and videoInputSelected, but it did not fix the issue. Your demo doesn't do that either.

evangeline commented 1 year ago

Realised the last log does not contain the audioSendingFailed event at the end, I have attached a new log that does. audio_sending_failed_21_Jul.txt

devalevenkatesh commented 1 year ago

I checked the latest attached logs, now, I do not see sendingAudioFailed as part of meeting join as I could see the meeting started fine and startAudioInput was successful before meeting start was attempted.

Couple of questions:

Could you once try doing startAudioInput before startVideoInput and see whether that helps?
If you do not join muted and un-mute later, do you still run into this issue? We have had issues in the past with muted join but more on Safari, not in Chrome I think.
The media stream that is selected, it has audio tracks, is enabled and muted is false right?
With this approach, does the audio recover or it keeps sending sendingAudioFailed?

evangeline commented 1 year ago

When the meeting join first occurs, and since the audio device is selected, is the attendee heard fine?

No, the attendee is never heard

Could you once try doing startAudioInput before startVideoInput and see whether that helps?

When joining the meeting, I start audio input with the default audio device before video input with the default video device. When both audio and video tracks are received, I also start the audio input first. This reduces SendingAudioFailed quite significantly but it still happens occasionally. Attached sending_audio_failed_21_Jul_2.log here. audio_sending_failed_21_Jul_2.log

If you do not join muted and un-mute later, do you still run into this issue? We have had issues in the past with muted join but more on Safari, not in Chrome I think.

Yes it still happens.

The media stream that is selected, it has audio tracks, is enabled and muted is false right?

Yes

With this tried approach, does the audio recover or it keeps sending sendingAudioFailed?

We always get sendingAudioFailed once. The audio does not recover, we usually have to call startAudioInput again (with the tracks cloned in a new mediastream) and audio always recovers after.

aws / amazon-chime-sdk-js