Custom Audio Devices

[aaalaniz]

Description

When creating a LocalAudioTrack, the Video Android SDK uses a default audio device that captures audio frames from the microphone. Unlike the Video iOS SDK, the Video Android SDK does not support providing your own audio device. As a developer, providing your own audio device would allow use cases where LocalAudioTracks could produce audio frames from sources other than the microphone, such as a file.

Our team is aware of this feature gap between Video iOS and Video Android. We can provide no timeline on when this feature will be available for Video Android developers, but any updates will posted in this issue.

[ceggert87]

Hey @aaalaniz - Just curious if there has been any update on this?

I just dived into the topic myself (though my task was "only" to change the audio device and not injecting files or anything like that) and created a topic in discuss-webrtc which got a developer response: https://groups.google.com/forum/#!topic/discuss-webrtc/G3uL89ZE2yM

The important part for me was being able to change the microphone that is accessed while still keeping the output device the same. That way, when putting e.g. a headset in which has a horrible microphone, the built-in microphone can still be used.

I'm not sure how the Twilio implementation looks like specifically for this, but for my own WebRTC it was quite easy to accomplish as I basically only needed to add a getter/setter to the JavaAudioDeviceModule/WebRtcAudioRecord classes and adding a audioRecord.setPreferredAudioDevice before it starts the recording. Works like a charm with the default implementations of WebRTC.

Just wondering if there's anything in the pipeline to have this in the Twilio API as well? The ticket sounds a bit more complex, so maybe I'm in the wrong spot here since files are mentioned. I'd love to have this solution when using Twilio as well :).

Cheers!

[acrolink]

Any news on this?

[aaalaniz]

Hey @ceggert87 and others,

We are actively working on adding custom audio device support. We are currently prioritizing the feature with our Voice Android SDK, but we intend to add this feature to the Video Android SDK as well. Stay tuned for updates to the Voice Android SDK for details about how this feature will work because it will be a very similar API with the Video Android SDK.

Thanks to all for the patience! 😄

[acrolink]

@aaalaniz I am happy to hear that. The issue is that I need to deliver the project in 7 days! I WANT to use Twilio and only Twilio.

For now I have two options, use Twilio for the Video and OpenTok for the voice or you could possibly guide me how to override: MediaRecorder.AudioSource.VOICE_COMMUNICATION and use instead of it: AudioSource.MIC.

Which Twilio source .java file I need to change?

Thank you.

[acrolink]

Seems that the VOICE_COMMUNICATION as Audio Source setting is hard coded inside the twillio_.so file. Correct me if I am wrong pleae.

[aaalaniz]

Hey @acrolink

You can give this snippet a try for changing the default audio source. I don't know how this will behave with our SDK, but I did validate that this code will change the value.

    private static void updateDefaultAudioSource() {
        Field field = null;
        try {
            field = WebRtcAudioRecord.class.getDeclaredField("DEFAULT_AUDIO_SOURCE");
            field.setAccessible(true);
            field.set(null, MediaRecorder.AudioSource.MIC);
            int defaultAudioSource = (int) field.get(null);
            Log.d("WebRtcAudioRecord", "DEFAULT_AUDIO_SOURCE: " + defaultAudioSource);
        } catch (IllegalAccessException e) {
            e.printStackTrace();
        } catch (NoSuchFieldException e) {
            e.printStackTrace();
        }
    }

[acrolink]

@aaalaniz

Thank you, I have tried the above code in my VideoActivity.java file (the one initiating the Twilio client). I have called the method you mentioned early in onCreate(). I see that it changes the value to MIC but the Twilio client still tries to initiate the AudioRecord by VOICE_COMMUNICATION (int value 7):

See the logs (pay attention to record source 7 in first line):

2020-03-28 00:36:31.206 19167-19194/com.twilio.video.quickstart E/AudioRecord: Could not get audio input for session 1001, record source 7, sample rate 48000, format 0x1, channel mask 0x10, flags 0
2020-03-28 00:36:31.208 19167-19194/com.twilio.video.quickstart E/AudioRecord-JNI: Error creating AudioRecord instance: initialization check failed with status -22.
2020-03-28 00:36:31.208 19167-19194/com.twilio.video.quickstart E/android.media.AudioRecord: Error code -20 when initializing native AudioRecord object.
2020-03-28 00:36:31.209 19167-19194/com.twilio.video.quickstart E/tvi.webrtc.Logging: WebRtcAudioRecord: Init recording error: Failed to create a new AudioRecord instance
2020-03-28 00:36:31.247 1894-2208/? E/PrimaryDeviceHAL: mmvsDevice not existed,Don't openmvsInputStream!!
2020-03-28 00:36:31.247 19167-19194/com.twilio.video.quickstart E/AudioRecord: Could not get audio input for session 1009, record source 7, sample rate 48000, format 0x1, channel mask 0x10, flags 0
2020-03-28 00:36:31.247 19167-19194/com.twilio.video.quickstart E/AudioRecord-JNI: Error creating AudioRecord instance: initialization check failed with status -22.
2020-03-28 00:36:31.247 19167-19194/com.twilio.video.quickstart E/android.media.AudioRecord: Error code -20 when initializing native AudioRecord object.
2020-03-28 00:36:31.248 19167-19194/com.twilio.video.quickstart E/tvi.webrtc.Logging: WebRtcAudioRecord: Init recording error: Failed to create a new AudioRecord instance
2020-03-28 00:36:31.462 1894-3386/? E/AUDIO: [Utopia]MDrv_AUDIO_PCM_Open: Open PCM device
2020-03-28 00:36:31.462 1894-3386/? E/AUDIO: [Utopia]HAL_AUDIO_PCM_HwDma_Reader1_Open: Open HW DMA Reader1
2020-03-28 00:36:31.462 1894-3386/? E/AUDIO: [Utopia]MDrv_AUDIO_PCM_Open: return device-id 0
2020-03-28 00:36:31.462 1894-3386/? E/AUDIO: [Utopia]MDrv_AUDIO_PCM_Start: Start PCM device-id 0
2020-03-28 00:36:31.462 1894-3386/? E/AUDIO: [Utopia]HAL_AUDIO_PCM_HwDma_Reader1_Start: Start HW DMA Reader1
2020-03-28 00:36:31.463 1894-3386/? E/AUDIO: [Utopia]HAL_AUDIO_PCM_HwDma_Reader1_Write: Warning! PCM buffer empty!
2020-03-28 00:36:31.463 1894-3386/? E/AUDIO: [Utopia]HAL_AUDIO_PCM_HwDma_Reader1_Stop: Stop HW DMA Reader1
2020-03-28 00:36:31.463 1894-3386/? E/AUDIO: [Utopia]HAL_AUDIO_PCM_HwDma_Reader1_Start: Start HW DMA Reader1
2020-03-28 00:36:31.485 1894-3386/? E/AUDIO: [Utopia]HAL_AUDIO_PCM_HwDma_Reader1_Write: Warning! PCM buffer empty!
2020-03-28 00:36:31.486 1894-3386/? E/AUDIO: [Utopia]HAL_AUDIO_PCM_HwDma_Reader1_Stop: Stop HW DMA Reader1
2020-03-28 00:36:31.486 1894-3386/? E/AUDIO: [Utopia]HAL_AUDIO_PCM_HwDma_Reader1_Start: Start HW DMA Reader1
2020-03-28 00:36:32.699 1905-1905/? E/HWComposer: Invalid layer type change: SolidColor --> Device

Should I try to put the code above inside the tvi.webrtc source files and reassemble the .aar ?

[aaalaniz]

Hey @acrolink

Two things.

1. I just discovered an API that does not require reflection WebRtcAudioRecord.setAudioSource(MediaRecorder.AudioSource.MIC); 🤦‍♂ (sorry i missed that)

2. Can you post a complete log? Thanks!

[acrolink]

@aaalaniz

TASK DONE! It worked.

WebRtcAudioRecord.setAudioSource(MediaRecorder.AudioSource.MIC);

Is the way to change it. It did it and now the MIC sound gets delivered to the other room participant.

Thank you very much. I wish you all well.

[aaalaniz]

Great news @acrolink! I'm so glad I could help. Do you mind describing your use case in more detail? I'd like to update our documentation accordingly if other developers can benefit from this API.

Thanks!

[acrolink]

@aaalaniz I am using an USB camera with an Android device and a special library to access it (saki4510t/UVCCamera). Using this library the webcam audio becomes available as AudioSource.MIC.

[vismay11]

Is this available in the Android SDK? Can we have a list of microphones available when the earphone with microphone is connected?

[aaalaniz]

Hey @vismay11

We recently released AudioSwitch. AudioSwitch helps developers manage audio devices in their real-time communications applications built with our Voice and Video SDKs. I recommend checking out the usage in the Quickstart and the Video Android Collaboration App.

Thanks!

[vismay11]

Hi @aaalaniz I checked Audio Switch but I want to have the list of available microphones and not audio output devices such as speaker,earphones etc. Is it possible in Audio Switch now? I went through your Readme just now and it's not mentioned anywhere. Can you please guide.

[vismay11]

@aaalaniz if you check the first comment by @ceggert87 in that the second paragraph describes a perfect case where it will be useful.

[aaalaniz]

Hey @vismay11

Thank you for the clarification. I see that what you are requesting is an entirely different feature from what AudioSwitch and Custom Audio Devices provide. Do you mind cutting a separate feature request issue so that the work can be tracked separately?

Thank you

[vismay11]

Hi @aaalaniz Sure will do that just wanted to know shall I put the future request here or in the AudioSwitch library?

[aaalaniz]

Hey @vismay11

This repo is fine. I don’t think this is an audio switch feature. Thank you!

[elagarrigue]

@aaalaniz Is there any way to send DTMF tones using the local audio track? There seems to be an issue in the backend to regarding sending DTMF tones to a room (call disconnects) and we are looking for a workaround from the clients.

[aaalaniz]

Hey @elagarrigue

There is currently no way to provide a DTMF tone via a LocalAudioTrack on Android. We are currently working on implementing custom audio devices which would enable implementing this functionality.

Thank you

[kbagchiGWC]

The Video Android SDK added AudioDevice API in 5.12.0 release.

See this example and API docs to learn more about custom audio devices.